{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "UMqBL77hMXP2"
      },
      "source": [
        "# Cost efficient prompting for **RAG**-based tasks: Applying compression to retrieved context for token count reduction  \n",
        "\n",
        "Authors:  \n",
        " - [Lior Gazit](https://www.linkedin.com/in/liorgazit).  \n",
        " - [Meysam Ghaffari](https://www.linkedin.com/in/meysam-ghaffari-ph-d-a2553088/).  \n",
        "\n",
        "This notebook is taught and reviewed in our book:  \n",
        "**[Mastering NLP from Foundations to LLMs](https://www.amazon.com/dp/1804619183)**  \n",
        "![image.png]()\n",
        "\n",
        "This Colab notebook is referenced in our book's Github repo:   \n",
        "https://github.com/PacktPublishing/Mastering-NLP-from-Foundations-to-LLMs   \n",
        "<a target=\"_blank\" href=\"https://colab.research.google.com/github/PacktPublishing/Mastering-NLP-from-Foundations-to-LLMs/blob/liors_branch/Chapter9_notebooks/Ch9_RAGLlamaIndex_Prompt_Compression.ipynb\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
        "</a>"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "LafJ-pq-ixA_"
      },
      "source": [
        "**The motivation for this notebook:**  \n",
        "We set up this notebook so to serve as a study for the impact of using content compression for the RAG's retrieved context text.  \n",
        "Typically, when applying the RAG scheme, the vast majority of the prompt's tokens are due to the context portion, meaning, the content that the RAG retrieved from the database to serve as the context for the LLM to refer to when performing the requested task.  \n",
        "We examine the use of [**LongLLMLingua**](https://arxiv.org/abs/2310.06839), which aims at identifying low information density in long contextand apply prompt compression. By doing so, it reduces the number of sent tokens while attempting to maintain all the relevant pieces of information for answering the user's prompt.  \n",
        "Not that the compression process is done with regards to the user's prompt.  \n",
        "Moreover, the compression process is done on a free local LLM, Llama2, thus not increasing the API cost.  \n",
        "\n",
        "**The description of the experiments:**  \n",
        "Each experiment represents a user setting a RAG task for asking a question about a particular Arxiv publication.  \n",
        "In each experiment, a single random Arxiv publication, which is categorized as an AI publication, is chosen. The user asks whether the publication's topic is around Reinforcement Learning, thus making the prompt be a request for a binary classification based on the publication's content. The LLM that we chose for performing this task is OpenAI's GPT.  \n",
        "Now, since each publication is very long when compared to the maximal allowed prompt length, a RAG is applied. Each publication is chunked and filed in a vector database.\n",
        "In each experiment, two approaches are employed:  \n",
        "1. Context for the prompt is being retrieved, gets attached to the user's question, and sent to OpenAI's GPT for classification,  \n",
        "1. Context for the prompt is being retrieved, **gets compressed by LongLLMLingua**, gets attached to the user's question, and sent to OpenAI's GPT for classification.  \n",
        "The following metrics are measured for each of these two approaches:  \n",
        "* Number of tokens sent to OpenAI's GPT  \n",
        "* Processing time  \n",
        "* Agreement/Disagreement in classification between the two  \n",
        "\n",
        "**Outcome:**  \n",
        "The notebook sums the study by presenting the trade offs for utilizing LongLLMLingua.  \n",
        "\n",
        "See notebooke **Ch9_Completing_a_Complex_Analysis_with_a_Team_of_LLM_Agents.ipynb** as we employ a team of agents to finalize the results from this analysis.  \n",
        "\n",
        "**Requirements:**  \n",
        "* When running in Colab, use this runtime notebook setting: `Python 3, T4 GPU`  \n",
        "* This code picks OpenAI's API as a choice of LLM, so a paid **API key** is necessary.   "
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "g54Uf66Vz9Fi"
      },
      "source": [
        ">*```Disclaimer: The content and ideas presented in this notebook are solely those of the authors and do not represent the views or intellectual property of the authors' employers.```*"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "H1K7R2ka5aWR"
      },
      "source": [
        "Install:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a970a901-11bd-43af-a8bc-7fb2fc6a1a07",
        "outputId": "e5baa5c4-c6aa-4700-963e-95576a51d311"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m280.0/280.0 kB\u001b[0m \u001b[31m8.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m536.7/536.7 kB\u001b[0m \u001b[31m46.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m15.3/15.3 MB\u001b[0m \u001b[31m76.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m86.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.8/1.8 MB\u001b[0m \u001b[31m93.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m20.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m21.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.6/75.6 kB\u001b[0m \u001b[31m12.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m108.0/108.0 kB\u001b[0m \u001b[31m14.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m226.7/226.7 kB\u001b[0m \u001b[31m28.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.4/4.4 MB\u001b[0m \u001b[31m68.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m284.0/284.0 kB\u001b[0m \u001b[31m32.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.8/77.8 kB\u001b[0m \u001b[31m12.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m8.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m30.6/30.6 MB\u001b[0m \u001b[31m53.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.4/49.4 kB\u001b[0m \u001b[31m7.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ],
      "source": [
        "# REMARK:\n",
        "# If the below code error's out due to a Python package discrepency, it may be because new versions are causing it.\n",
        "# In which case, set \"default_installations\" to False to revert to the original image:\n",
        "default_installations = True\n",
        "if default_installations:\n",
        "  !pip -q install llmlingua llama-index==0.10.7 accelerate datasets llama-index-postprocessor-longllmlingua\n",
        "else:\n",
        "  import requests\n",
        "  text_file_path = \"requirements__Ch9_RAGLlamaIndex_Prompt_Compression.txt\"\n",
        "  url = \"https://raw.githubusercontent.com/PacktPublishing/Mastering-NLP-from-Foundations-to-LLMs/main/Chapter9_notebooks/\" + text_file_path\n",
        "  res = requests.get(url)\n",
        "  with open(text_file_path, \"w\") as f:\n",
        "    f.write(res.text)\n",
        "\n",
        "  !pip install -r requirements__Ch9_RAGLlamaIndex_Prompt_Compression.txt"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "MqNJyqCC5cEm"
      },
      "source": [
        "Import:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3_nBfgro6BkP"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import gc\n",
        "import openai\n",
        "from datasets import load_dataset\n",
        "import pandas as pd\n",
        "from llama_index.core import Document\n",
        "from llama_index.core import VectorStoreIndex\n",
        "from llama_index.llms.openai import OpenAI\n",
        "from llama_index.core.indices.query.schema import QueryBundle\n",
        "from llama_index.core.response_synthesizers import CompactAndRefine\n",
        "from llama_index.postprocessor.longllmlingua import LongLLMLinguaPostprocessor\n",
        "import random\n",
        "import time\n",
        "import pickle\n",
        "import os"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Cljc_GWq5o9H"
      },
      "source": [
        "### Code Settings"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lQ36ZrE5Qqiy"
      },
      "outputs": [],
      "source": [
        "# This sets the prompted model type for the non-compressing RAG (the prompted\n",
        "# model for the compressing RAG is set by default to \"gpt-3.5-turbo\", while\n",
        "# the compressing model is set by default to \"NousResearch/Llama-2-7b-hf\")\n",
        "gpt_type = \"gpt-3.5-turbo\"\n",
        "\n",
        "# Number of experiments to run. Each experiment represents a user setting a RAG task for asking a question about a particular Arxiv publication:\n",
        "num_of_iterations = 60\n",
        "\n",
        "# As the Arxiv publication for each experiment is chosen at random, we set a fixed seed so to allow reproducibility of the original results:\n",
        "set_seed = 0\n",
        "\n",
        "# Number of similar chunks to retrieve by the RAG for each experiment:\n",
        "similarity_top_k = 5\n",
        "\n",
        "# A targen amount of tokens to aim to reduce to when compressing the retrieved context:\n",
        "target_token = 2**9"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eFM9sYq06J1f"
      },
      "outputs": [],
      "source": [
        "pd.set_option('display.max_colwidth', None)\n",
        "pd.set_option('display.max_rows', None)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "lPpqKBtA5jtM"
      },
      "source": [
        "Define OpenAI's API key:  \n",
        "**You must provide a key and paste it as a string!**  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cbbbf3de-a9d6-46cf-afab-dcb72a6154ec"
      },
      "outputs": [],
      "source": [
        "openai.api_key = \"...\""
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "f8676ffa-5117-44dc-9742-bb9ab1d56e0c"
      },
      "source": [
        "## Gather the Data\n",
        "The data is taken from here:  \n",
        "https://huggingface.co/datasets/ccdv/arxiv-classification\n",
        "\n",
        "It contains content of full [Arxiv](https://arxiv.org/) publications.  \n",
        "We will focus on the AI category.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 466,
          "referenced_widgets": [
            "926d325278334e63ae32ecd81c76ed9c",
            "8998660271044d47b3d04f0b623bd280",
            "9a593469841c4ad798590b092d48184a",
            "4d6afa19e5d34fa7b74f1dc9ee525e2a",
            "53207f366c654487ae2298fd98de8958",
            "3eae91e857b64b299649f877c70d6eb4",
            "808ad1d7b29949969d3aa4ea6ad33105",
            "32f6a39d5eef4977a260fc42d0131b15",
            "bcf0b1dc6aa44f0ba60865191490ad6a",
            "81972485b98040389eb1f99309ea731d",
            "91f3807f41ea459981b6c099d76204e1",
            "1e260394422842bfb8abdcc228e52950",
            "9e4b2a702825461a9cd1c2efc3028764",
            "5e6ac0a353514669825fe946c20036fc",
            "c8d2af6aa9ca43628b344f6bd488878d",
            "85bc1552ef7d4108959ae8f5a638993c",
            "90aec7e6cf684a06aea6e48653fa0512",
            "6b357bc0c2f6415085c497ff50e703bc",
            "775933ae7b6c40528669a34b25f6eb2d",
            "c6ca3ceaaa194f0c80a6712f3c57bfc8",
            "8615b76ba082478ba99e1d4cddf1b0b2",
            "e35c9154ed09420988dfeb729f17368c",
            "63f50891984640139a7eda23892b35f1",
            "ac8bec4924ea491e8c7629fec7e725a3",
            "8e05142f69e347b0ab6ba16d3cc09cf1",
            "3516d6064921470abf3bf20f4a217d9e",
            "854fc31fd2734a96bd5476503228ccef",
            "5598ea77bad94eecbcaff771dd0ae52f",
            "e1568699381d410fb5112cfb788ba6a4",
            "bea914d2e8db47c7ba3c245a0b3b5baa",
            "1491d6d8b5154e95812c5c971c3b4eee",
            "2aa538f3cc2541cb970dd9bd2f9bb523",
            "3470c362d3dc4b6e96287fcf0610b4ef",
            "6fe8ffee3ea44d7b8f7d74669be4a737",
            "690b3df3217d4e8fa703818a1ae6015d",
            "00f247c633f244bf9c692ca8262920c8",
            "37d75872325f4dc18e0704ae09950429",
            "5bc8736baf6c4ce988f13e432b4f4be6",
            "0bf9e2088bb14aeba3bc6e2c3441a768",
            "34cd9b30041c471b8948b110ef6450f1",
            "380bfec326bf4cf6ab8c48830b7e9b1c",
            "0dbfeb79356d466c934748c754d59400",
            "fe75bbaa772644ca8965c34c59eee8c0",
            "221abbda138145cd8e46e764dc9b3860",
            "e74052b5d9f54b69838fb4e0de72eccd",
            "7f0579d35ce943288f79711d07ba3129",
            "cc46e93ca07d42bab9ae9c4bb14e706d",
            "70cfb580763d498a869cdcf259a1cd96",
            "944fe5ee5d8b4fd69df3bf1900d1a578",
            "5dff1bfeee5148a38580ad5d676af044",
            "e2388d0ffd9d44b181227996e40f1ce8",
            "95cd0639a0ae4f10b445a2b9d6885d4a",
            "249359bdbfc142d5b722053b820fef52",
            "41585a2384394425b0c6dc9e694db127",
            "de15f73ec9884a3ca2758b378d40c6f0",
            "2a409b6ba29e4e1aba794f452f814bde",
            "d2ea697878524f34bef022b9b632a775",
            "f559913d5ef0411683357a5ca50885b5",
            "4f5b1c35348c464a8697a34a770f5ffb",
            "b6b75247cdf44f76ba775dc3238b3f5a",
            "651e126cd06847599331d530fc907c25",
            "2dc2c29422c3437b8859be6598bd6adc",
            "b2e6c1a987104f1cbab55a1cf965c369",
            "de97f327e21a42e4b077f393f5195183",
            "82a954c42f3a402c826ab7a473a42cff",
            "a05e76f8351c42cb9aa9ab72f78f31fa",
            "83e553d816334b8b8019e05130c0abed",
            "59173acf03c646f4ab987a27e6a62127",
            "9e8f8ae8e9b34e5b95adeded355cdae8",
            "7a297bb903904363ac481154e9c8600c",
            "cf595012852e49e2a538d7cc65c769c5",
            "d8261f3ed03b4999a006b531e78619fc",
            "4e4548e7d87845b8a51e4da9c57d1174",
            "4cb1db48b6f14bfcbd112b4865e4b287",
            "c78397492dfd4bae8dfcd0f26a7e5594",
            "8c79455638c5477fbe25eb73bc79b886",
            "e8fead7dfc0a4a33a8e2c1e679e32be6",
            "8a2d538ace784e62896070d0d5ebf354",
            "e2677b3b568d46e1aaddf20f4b4bf9ad",
            "4d2d13b784e14adb9359bc6c71cd613f",
            "acd6967ac6e341f9afe670aa91a7a31e",
            "dbc39de7bee64a07b04a013b49a484d3",
            "adeb62cdecb649c0b81ed530f0dc58cb",
            "87526647f4e04a1d82bc961e72b26d86",
            "641f9c94ab244fa0b3a4153953976336",
            "66b62850c8384e12b16005798149dd73",
            "d915b903dffd47cda0e9fa95907859da",
            "42daf8300eab49ec9a8040d3b0d9ebcf"
          ]
        },
        "id": "bb349566-83d8-44ac-a683-b67ed9ddf7a6",
        "outputId": "55a8351d-32d2-45d4-8016-ff38f21d0941"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:88: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/datasets/load.py:1454: FutureWarning: The repository for ccdv/arxiv-classification contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/ccdv/arxiv-classification\n",
            "You can avoid this message in future by passing the argument `trust_remote_code=True`.\n",
            "Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "926d325278334e63ae32ecd81c76ed9c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading builder script:   0%|          | 0.00/3.80k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "1e260394422842bfb8abdcc228e52950",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading readme:   0%|          | 0.00/1.68k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "63f50891984640139a7eda23892b35f1",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading data:   0%|          | 0.00/1.71G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6fe8ffee3ea44d7b8f7d74669be4a737",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading data:   0%|          | 0.00/150M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "e74052b5d9f54b69838fb4e0de72eccd",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading data:   0%|          | 0.00/146M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "2a409b6ba29e4e1aba794f452f814bde",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Generating train split: 0 examples [00:00, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "83e553d816334b8b8019e05130c0abed",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Generating validation split: 0 examples [00:00, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "8a2d538ace784e62896070d0d5ebf354",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Generating test split: 0 examples [00:00, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "dataset = load_dataset(\"ccdv/arxiv-classification\")\n",
        "dataset_train = dataset[\"train\"]\n",
        "dataset_train_df = pd.DataFrame(dataset_train)\n",
        "\n",
        "# Justin keeping the AI publications (categorized as #2):\n",
        "dataset_train_df_ai = dataset_train_df[dataset_train_df[\"label\"] == 2].reset_index(drop=True)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "tDKEBkAO-c7V"
      },
      "source": [
        "Observe the full content of one of the publications:  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "9VwS-s-mwr_e",
        "outputId": "637a3c4a-af4b-434e-ae61-eaf0137ce0b8"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<style type=\"text/css\">\n",
              "#T_ea679_row0_col0, #T_ea679_row0_col1 {\n",
              "  text-align: left;\n",
              "}\n",
              "</style>\n",
              "<table id=\"T_ea679\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th class=\"blank level0\" >&nbsp;</th>\n",
              "      <th id=\"T_ea679_level0_col0\" class=\"col_heading level0 col0\" >text</th>\n",
              "      <th id=\"T_ea679_level0_col1\" class=\"col_heading level0 col1\" >label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th id=\"T_ea679_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "      <td id=\"T_ea679_row0_col0\" class=\"data row0 col0\" >arXiv:1711.06299v1 [cs.LG] 16 Nov 2017\n",
              "\n",
              "Bayesian Best-Arm Identification for Selecting\n",
              "Influenza Mitigation Strategies\n",
              "Pieter Libin1,2 , Timothy Verstraeten1 , Diederik M. Roijers1 , Jelena\n",
              "Grujic1 , Kristof Theys2 , Philippe Lemey2 , and Ann Nowé1\n",
              "1\n",
              "\n",
              "Artificial Intelligence Lab, Department of computer science, Vrije\n",
              "Universiteit Brussel, Brussels, Belgium\n",
              "2\n",
              "KU Leuven University of Leuven, Department of Microbiology\n",
              "and Immunology, Rega Institute for Medical Research, Clinical\n",
              "and Epidemiological Virology, Leuven, Belgium\n",
              "November 20, 2017\n",
              "Abstract\n",
              "Pandemic influenza has the epidemic potential to kill millions of people. While various preventive measures exist (i.a., vaccination and school\n",
              "closures), deciding on strategies that lead to their most effective and efficient use, remains challenging. To this end, individual-based epidemiological models are essential to assist decision makers in determining the\n",
              "best strategy to curve epidemic spread. However, individual-based models are computationally intensive and therefore it is pivotal to identify\n",
              "the optimal strategy using a minimal amount of model evaluations. Additionally, as epidemiological modeling experiments need to be planned,\n",
              "a computational budget needs to be specified a priori. Consequently, we\n",
              "present a new sampling method to optimize the evaluation of preventive\n",
              "strategies using fixed budget best-arm identification algorithms. We use\n",
              "epidemiological modeling theory to derive knowledge about the reward\n",
              "distribution which we exploit using Bayesian best-arm identification algorithms (i.e., Top-two Thompson sampling and BayesGap). We evaluate\n",
              "these algorithms in a realistic experimental setting and demonstrate that\n",
              "it is possible to identify the optimal strategy using only a limited number\n",
              "of model evaluations, i.e., 2-to-3 times faster compared to the uniform\n",
              "sampling method, the predominant technique used for epidemiological\n",
              "decision making in the literature. Finally, we contribute and evaluate a\n",
              "statistic for Top-two Thompson sampling to inform the decision makers\n",
              "about the confidence of an arm recommendation.\n",
              "\n",
              "1\n",
              "\n",
              "\f1\n",
              "\n",
              "Introduction\n",
              "\n",
              "The influenza virus is responsible for the deaths of half of a million people each\n",
              "year [52]. In addition, seasonal influenza epidemics cause a significant economic\n",
              "burden [45]. While transmission is primarily local, a newly emerging variant may\n",
              "spread to pandemic proportions in a naive (i.e., fully susceptible) host population [46]. Pandemic influenza occurs less frequently than seasonal influenza\n",
              "but the outcome with respect to morbidity and mortality can be much more\n",
              "severe, potentially killing millions of people worldwide [47, 48]. Consequently,\n",
              "it is essential to study mitigation strategies to control influenza pandemics.\n",
              "For influenza, different preventive measures exist: i.a., vaccination, social\n",
              "measures (e.g. school closures and travel restrictions) and antiviral drugs. However, the efficiency of strategies greatly depends on the availability of preventive\n",
              "compounds, as well as on the characteristics of the targeted epidemic. Furthermore, governments typically have limited resources to implement such measures.\n",
              "Therefore, it remains challenging to formulate public health strategies that make\n",
              "effective and efficient use of these preventive measures within the existing resource constraints.\n",
              "Epidemiological models (i.e., compartment models and individual-based models) are essential to study the effects of preventive measures in silico [8, 27].\n",
              "While individual-based models are usually associated with a greater model complexity and computational cost than compartment models, they allow for a more\n",
              "accurate evaluation of preventive strategies [12, 19, 44]. To capitalize on these\n",
              "advantages and make it feasible to employ individual-based models, it is essential to use the available computational resources as efficiently as possible.\n",
              "In the literature, a set of possible preventive strategies is typically evaluated\n",
              "by simulating each of the strategies an equal number of times [25, 5, 30, 41, 20,\n",
              "22, 13]. However, this approach is inefficient to identify the optimal preventive\n",
              "strategy, as a large proportion of computational resources will be used to explore\n",
              "sub-optimal strategies. Furthermore, a consensus on the required number of\n",
              "model evaluations per strategy is currently lacking [57]. Moreover, as we show\n",
              "in this paper, this number depends on the hardness of the evaluation problem\n",
              "[6].\n",
              "For this reason, we propose to combine individual-based epidemiological\n",
              "models with multi-armed bandits [32]. In a preliminary study [39], the potential\n",
              "of multi-armed bandits was explored in a regret minimization setting, using default strategies (i.e., ǫ-greedy [53] and UCB1 [7]). However, in this work, we recognize that epidemiological modeling experiments need to be planned and that\n",
              "a computational budget needs to be specified a priori. Within this constraint,\n",
              "we aim to minimize the number of required model evaluations to determine the\n",
              "most promising preventive strategy. Therefore, we present a novel approach\n",
              "formulating the evaluation of preventive strategies as a best-arm identification\n",
              "problem using a fixed budget of model evaluations.\n",
              "As running an individual-based model is computationally intensive (i.e., minutes to hours, depending on the complexity of the model), minimizing the number of required model evaluations reduces the total time required to evaluate\n",
              "2\n",
              "\n",
              "\fa given set of preventive strategies. This renders the use of individual-based\n",
              "models attainable in studies where it would otherwise not be computationally\n",
              "feasible. Additionally, reducing the number of model evaluations can also free\n",
              "up computational resources in studies that already use individual-based models, capacitating researchers to explore different model scenarios. Considering\n",
              "a wider range of scenarios increases the confidence about the overall utility of\n",
              "preventive strategies [58].\n",
              "In our model, an arm’s reward distribution corresponds to the epidemic size\n",
              "distribution of the epidemiological model. We employ epidemiological modeling\n",
              "theory to derive that this distribution is approximately Gaussian, and exploit\n",
              "this knowledge using Bayesian best-arm identification algorithms.\n",
              "In this paper, we contribute a novel method to evaluate preventive strategies\n",
              "as a best-arm identification problem. This method enables decision makers to\n",
              "obtain recommendations in a reduced number of model evaluations and supports\n",
              "their decision process by providing a confidence recommendation statistic. In\n",
              "Section 4 we employ concepts from epidemiological model theory and we adapt\n",
              "Bayesian best-arm identification algorithms to incorporate this knowledge. In\n",
              "Section 5, we evaluate these algorithms in an experimental setting, where we aim\n",
              "to find the best vaccine allocation strategy in a realistic simulation environment\n",
              "that models Seattle’s social network. We repeat the experiment for a wide\n",
              "range of basic reproduction numbers (i.e., R0 , the number of infections that\n",
              "is, by average, generated by one single infection) that are typically used in the\n",
              "influenza literature. The obtained experimental results show that our approach\n",
              "is able to identify the best preventive strategy 2-to-3 times faster compared to\n",
              "uniform sampling, the predominant technique used for epidemiological decision\n",
              "making in the literature. Furthermore, we contribute (Section 4) and evaluate\n",
              "(Section 5) a statistic to inform the decision makers about the confidence of a\n",
              "particular recommendation.\n",
              "\n",
              "2\n",
              "2.1\n",
              "\n",
              "Background\n",
              "Pandemic influenza and vaccine production\n",
              "\n",
              "The primary preventive strategy to mitigate seasonal influenza is to produce\n",
              "vaccine prior to the epidemic, anticipating the virus strains that are expected\n",
              "to circulate. This vaccine pool is used to inoculate the population before the\n",
              "start of the epidemic. While seasonal influenza may have a restricted susceptible\n",
              "population due to vaccination and pre-existing immunity, a newly emerging\n",
              "strain can become pandemic by spreading rapidly among naive human hosts\n",
              "worldwide [46].\n",
              "While it is possible to stockpile vaccines to prepare for seasonal influenza,\n",
              "this is not the case for new variants of influenza viruses, as the vaccine should be\n",
              "specifically tailored to the virus that is the source of the pandemic. Therefore,\n",
              "before an appropriate vaccine can be produced, the responsible virus needs to\n",
              "be identified. Hence, vaccine will be available only in limited supply at the\n",
              "\n",
              "3\n",
              "\n",
              "\fbeginning of the pandemic [56]. In addition, production problems can result\n",
              "in vaccine shortages [18]. When the number of vaccine doses is limited, it is\n",
              "imperative to identify an optimal vaccine allocation strategy [43].\n",
              "\n",
              "2.2\n",
              "\n",
              "Modeling influenza\n",
              "\n",
              "There is a long tradition to use individual-based models to study influenza\n",
              "epidemics [8, 27, 25], as it allows for a more accurate evaluation of preventive\n",
              "strategies. A state-of-the-art individual-based model, that has been the driver\n",
              "for many high impact research efforts [8, 27, 29], is FluTE [12].\n",
              "FluTE implements a contact model where the population is divided into communities of households [12]. The population is organized in a hierarchy of social\n",
              "mixing groups where the contact intensity is inversely proportional with the size\n",
              "of the group (e.g., closer contact between members of a household than between\n",
              "colleagues). Additionally, FluTE implements an individual disease progression\n",
              "model, that associates different disease stages with different levels of infectiousness. To support the evaluation of preventive strategies, FluTE implements\n",
              "the simulation of therapeutic interventions (i.e., vaccines, antiviral compounds)\n",
              "and non-therapeutic interventions (i.e., school closure, case isolation, household\n",
              "quarantine).\n",
              "\n",
              "2.3\n",
              "\n",
              "Bandits and best-arm identification\n",
              "\n",
              "The multi-armed bandit game [6] concerns a K-armed bandit (i.e., a slot machine\n",
              "with K levers), where each arm Ak returns a reward rk when it is pulled (i.e.,\n",
              "rk represents a sample from Ak ’s reward distribution). A common use of the\n",
              "bandit game is to pull a sequence of arms such that the cumulative regret\n",
              "is minimized [32]. To fulfill this goal, the player needs to carefully balance\n",
              "between exploitation (i.e., choose the arms with the highest expected reward)\n",
              "and exploration (i.e., explore the other arms to potentially identify even more\n",
              "promising arms).\n",
              "In this paper, the objective is to recommend the best arm A∗ (i.e., the arm\n",
              "with the highest average reward µ∗ ) after a fixed number of arm pulls. This is\n",
              "referred to as the fixed budget best-arm identification problem [6], an instance\n",
              "of the pure-exploration problem [10]. For a given budget T , the objective is\n",
              "to minimize the simple regret µ∗ − µJ , where µJ is the average reward of the\n",
              "recommended arm AJ at time T [11]. Simple regret is inversely proportional to\n",
              "the probability of recommending the correct arm A∗ [38].\n",
              "\n",
              "3\n",
              "\n",
              "Related work\n",
              "\n",
              "In this work, we recognize that a computational budget needs to be specified\n",
              "a priori to meet the realities associated with high performance computational\n",
              "infrastructure. For this reason we consider the fixed budget best-arm identification setting, in contrast to techniques that attempt to identify the best arm with\n",
              "\n",
              "4\n",
              "\n",
              "\fa predefined confidence: i.e., racing strategies [21], strategies that exploit the\n",
              "confidence bound of the arms’ means [37, 35] and more recently fixed confidence\n",
              "best-arm identification algorithms [26].\n",
              "While other algorithms exist to rank or select bandit arms, e.g. [49], bestarm identification is best approached using adaptive sampling methods [36], as\n",
              "the ones we study in this paper. Moreover, the use of best-arm identification\n",
              "methods clears the way for interesting future work with respect to evaluating\n",
              "preventive strategies while considering multiple objectives (see Section 6).\n",
              "\n",
              "4\n",
              "\n",
              "Methods\n",
              "\n",
              "We formulate the evaluation of preventive strategies as a multi-armed bandit\n",
              "game with the aim of identifying the best arm using a fixed budget of model evaluations. The presented method is generic with respect to the type of epidemic\n",
              "that is modeled (i.e., pathogen, contact network, preventive strategies). The\n",
              "method is evaluated in the context of pandemic influenza in the next section.\n",
              "\n",
              "4.1\n",
              "\n",
              "Preventive bandits\n",
              "\n",
              "Definition 1. A stochastic epidemiological model E is defined in terms of a\n",
              "model configuration c ∈ C and can be used to evaluate a preventive strategy\n",
              "p ∈ P. Evaluating the model E results in a sample of the model’s outcome\n",
              "distribution:\n",
              "outcome ∼ E(c, p), where c ∈ C and p ∈ P\n",
              "(1)\n",
              "Note that a model configuration c ∈ C describes the complete model environment, i.e., both aspects inherent to the model (e.g., FluTE’s mixing model)\n",
              "and options that the modeler can provide (e.g., population statistics, vaccine\n",
              "properties). The result of a model evaluation is referred to as the model outcome\n",
              "(e.g., prevalence, proportion of symptomatic individuals, morbidity, mortality,\n",
              "societal cost).\n",
              "Our objective is to find the optimal preventive strategy from a set of alternative strategies {p1 , ..., pK } ⊂ P for a particular configuration c0 ∈ C of a\n",
              "stochastic epidemiological model, where c0 corresponds to the studied epidemic.\n",
              "Definition 2. A preventive bandit [39] has K = |{p1 , ..., pK }| arms. Pulling\n",
              "arm pk corresponds to evaluating pk by running a simulation in the epidemiological model E(c0 , pk ).\n",
              "A preventive bandit is thus a multi-armed bandit, that has preventive strategies as arms with reward distributions corresponding to the outcome distribution of a stochastic epidemiological model E(c0 , pk ). While the parameters of\n",
              "the reward distribution are known (i.e., the parameters of the epidemiological\n",
              "model), it is intractable to determine the optimal reward analytically from the\n",
              "epidemiological model. Hence, we must learn about the outcome distribution\n",
              "via interaction with the epidemiological model.\n",
              "\n",
              "5\n",
              "\n",
              "\f4.2\n",
              "\n",
              "Outcome distribution\n",
              "\n",
              "As previously defined, the reward distribution associated with a preventive bandit’s arm corresponds to the outcome distribution of the epidemiological model\n",
              "that is evaluated when pulling that arm. Therefore, employing insights from\n",
              "epidemiological modeling theory allows us to specify prior knowledge about the\n",
              "reward distribution.\n",
              "It is well known that a disease outbreak has two possible outcomes: either\n",
              "it is able to spread beyond a local context and becomes a fully established\n",
              "epidemic or it fades out [55]. Most stochastic epidemiological models reflect\n",
              "this reality and hence its epidemic size distribution is bimodal [55]. When\n",
              "evaluating preventive strategies, the objective is to determine the preventive\n",
              "strategy that is most suitable to mitigate an established epidemic. As in practice\n",
              "we can only observe and act on established epidemics, epidemics that faded\n",
              "out in simulation would bias this evaluation. Consequently, it is necessary to\n",
              "focus on the mode of the distribution that is associated with the established\n",
              "epidemic. Therefore we censor (i.e., discard) the epidemic sizes that correspond\n",
              "to the faded epidemic. The size distribution that remains (i.e., the one that\n",
              "corresponds with the established epidemic) is approximately Gaussian [9].\n",
              "In this study, we consider a scaled epidemic size distribution, i.e., the proportion of symptomatic infections. Hence we can assume bimodality of the full\n",
              "size distribution and an approximately Gaussian size distribution of the established epidemic. We verified experimentally that these assumptions hold for all\n",
              "the reward distributions that we observed in our experiments (see Section 5).\n",
              "To censor the size distribution, we use a threshold that represents the number\n",
              "of infectious individuals that are required to ensure an outbreak will only fade\n",
              "out with a low probability.\n",
              "\n",
              "4.3\n",
              "\n",
              "Epidemic fade-out threshold\n",
              "\n",
              "For heterogeneous host populations (i.e., a population with a significant variance\n",
              "among individual transmission rates, as is the case for influenza epidemics [16,\n",
              "24]), the number of secondary infections can be accurately modeled using a\n",
              "negative binomial offspring distribution NB(R0 , γ) [40], where R0 is the basic\n",
              "reproductive number and γ is a dispersion parameter that specifies the extent of\n",
              "heterogeneity. The probability of epidemic extinction pext can be computed by\n",
              "solving g(s) = s, where g(s) is the probability generating function (pgf) of the\n",
              "offspring distribution [40]. For an epidemic where individuals are targeted with\n",
              "preventive measures (i.e., vaccination in our use case), we obtain the following\n",
              "pgf\n",
              "\u0001−γ\n",
              "R0\n",
              "(1 − s)\n",
              "(2)\n",
              "g(s) = popc + (1 − popc ) 1 +\n",
              "γ\n",
              "\n",
              "where popc signifies the random proportion of controlled individuals [40]. From\n",
              "pext we can compute a threshold T0 to limit the probability of extinction to a\n",
              "cutoff ℓ [31].\n",
              "\n",
              "6\n",
              "\n",
              "\f4.4\n",
              "\n",
              "Best-arm identification with a fixed budget\n",
              "\n",
              "Our objective is to identify the best preventive strategy (i.e., the strategy that\n",
              "minimizes the expected outcome) out of a set of preventive strategies, for a\n",
              "particular configuration c0 ∈ C using a fixed budget T of model evaluations.\n",
              "Successive Rejects was the first algorithm to solve the best-arm identification\n",
              "in a fixed budget setting [6]. For a K-armed bandit, Successive Rejects operates\n",
              "in (K − 1) phases. At the end of each phase, the arm with the lowest average\n",
              "reward is discarded. Thus, at the end of phase (K − 1) only one arm survives,\n",
              "and this arm is recommended. At phase f ∈ {1, . . . , K − 1}, each arm that is\n",
              "still available is played mf − mf −1 times, where\n",
              "\n",
              "with\n",
              "\n",
              "m0 = 0\n",
              "\u0006 T −K\n",
              "1 \u0007\n",
              "mf =\n",
              "K + 1 − f log(K)\n",
              "\n",
              "(3)\n",
              "\n",
              "K\n",
              "\n",
              "log(K) =\n",
              "\n",
              "1 X1\n",
              "+\n",
              "2\n",
              "k\n",
              "\n",
              "(4)\n",
              "\n",
              "k=2\n",
              "\n",
              "Successive Rejects serves as a useful baseline, however, it has no support to\n",
              "incorporate any prior knowledge. Bayesian best-arm identification algorithms\n",
              "are able to take into account such knowledge by defining an appropriate prior\n",
              "and posterior on the arms’ reward distribution. As we will show, such prior\n",
              "knowledge can increase the best-arm identification accuracy. Additionally, at\n",
              "the time an arm is recommended, the posteriors contain valuable information\n",
              "that can be used to formulate a variety of statistics helpful to assist decision\n",
              "makers. We consider two state-of-the-art Bayesian algorithms: BayesGap [33]\n",
              "and Top-two Thompson sampling [51]. For Top-two Thompson sampling, we\n",
              "derive a statistic based on the posteriors to inform the decision makers about\n",
              "the confidence of an arm recommendation: the probability of success.\n",
              "As we established in the previous section, each arm of our preventive bandit\n",
              "has a reward distribution that is approximately Gaussian with unknown mean\n",
              "and variance. To make our method generic for any type of preventive bandit\n",
              "problem, we assume an uninformative Jeffreys prior (σk )−3 on (µk , σk2 ) [34].\n",
              "Honda and Takemura [34] demonstrate that this prior leads to the following\n",
              "posterior on µk at the nth\n",
              "k pull:\n",
              "s\n",
              "n2k\n",
              "(5)\n",
              "(µk − xk,nk ) | xk,nk , Sk,nk ∼ Tnk\n",
              "Sk,nk\n",
              "where xk,nk is the reward mean, Sk,nk is the sum of squares\n",
              "Sk,nk =\n",
              "\n",
              "nk\n",
              "X\n",
              "\n",
              "(rk,m − xk,nk )2\n",
              "\n",
              "m=1\n",
              "\n",
              "and Tnk is the standard student t-distribution with nk degrees of freedom.\n",
              "7\n",
              "\n",
              "(6)\n",
              "\n",
              "\fBayesGap is a gap-based Bayesian algorithm [33]. The algorithm requires\n",
              "that for each arm Ak , a high-probability upper bound Uk (t) and lower bound\n",
              "Lk (t) is defined on the posterior of µk at each time step t. Using these bounds,\n",
              "the gap quantity\n",
              "Bk (t) = max Ul (t) − Lk (t)\n",
              "(7)\n",
              "l6=k\n",
              "\n",
              "is defined for each arm Ak . Bk (t) represents an upper bound on the simple regret\n",
              "(as defined in Section 2.3). At each step t of the algorithm, the arm J(t) that\n",
              "minimizes the gap quantity Bk (t) is compared to the arm j(t) that maximizes\n",
              "the upper bound Uk (t). From J(t) and j(t), the arm with the highest confidence\n",
              "diameter Uk (t) − Lk (t) is pulled. The reward that results from this pull is\n",
              "observed and used to update Ak ’s posterior. When the budget is consumed, the\n",
              "arm\n",
              "J(argmin BJ(t) (t))\n",
              "(8)\n",
              "t≤T\n",
              "\n",
              "is recommended. This is the arm that minimizes the simple regret bound over\n",
              "all times t ≤ T .\n",
              "In order to use BayesGap in the preventive bandit setting, we contribute\n",
              "problem-specific bounds. Given our posteriors (Equation 5), we define\n",
              "Uk (t) = µ̂k (t) + β σ̂k (t)\n",
              "Lk (t) = µ̂k (t) − β σ̂k (t)\n",
              "\n",
              "(9)\n",
              "\n",
              "where µ̂k (t) and σ̂k (t) are the mean and standard deviation of the posterior of\n",
              "arm Ak at time step t, and β is the exploration coefficient.\n",
              "The amount of exploration that is feasible given a particular bandit game, is\n",
              "proportional to the available budget, and inversely proportional to the game’s\n",
              "complexity [33]. This complexity can be modeled taking into account the game’s\n",
              "hardness [6] and the variance of the rewards. Following Hoffman et al. [33], we\n",
              "define a hardness quantity\n",
              "X\n",
              "−2\n",
              "Hǫ =\n",
              "Hk,ǫ\n",
              "(10)\n",
              "k\n",
              "\n",
              "with arm-dependent hardness\n",
              "\n",
              "1\n",
              "Hk,ǫ = max( (∆k + ǫ), ǫ)\n",
              "2\n",
              "∆k = max(µl ) − µk\n",
              "\n",
              "(11)\n",
              "\n",
              "l6=k\n",
              "\n",
              "Considering the budget T , hardness Hǫ and a generalized reward variance\n",
              "2\n",
              "σG\n",
              "over all arms, we define\n",
              "s\n",
              "T − 3K\n",
              "(12)\n",
              "β=\n",
              "2\n",
              "4Hǫ σG\n",
              "Theorem 1 in the Supplementary Information1 formally proves that using these\n",
              "bounds results in a probability of simple regret that asymptotically reaches the\n",
              "exponential lower bound presented by Hoffman et al. [33].\n",
              "1 Supplementary\n",
              "\n",
              "Information is available at the end of this manuscript\n",
              "\n",
              "8\n",
              "\n",
              "\f2\n",
              "As both Hǫ and σG\n",
              "are unknown, in order to compute β, these quantities\n",
              "need to be estimated. Firstly, we estimate Hǫ ’s upper bound Ĥǫ by estimating\n",
              "∆k as follows\n",
              "\u0001\n",
              "\u0001\n",
              "ˆ k = max\n",
              "∆\n",
              "µ̂l (t) + 3σ̂l (t) − µ̂k (t) − 3σ̂k (t)\n",
              "(13)\n",
              "1≤l<K;l6=k\n",
              "\n",
              "2\n",
              "as in Hoffman et al. [33]. Secondly, for σG\n",
              "we need a measure of variance that\n",
              "is representative for the reward distribution of all arms. To this end, when the\n",
              "arms are initialized, we observe their sample variance s2k , and compute their\n",
              "average\n",
              "PK 2\n",
              "s\n",
              "2\n",
              "s̄G = k=1 k\n",
              "(14)\n",
              "K\n",
              "As our bounds depend on the standard deviation σ̂k (t) of the t-distributed\n",
              "posterior, each arm’s posterior needs to be initialized 3 times to ensure that\n",
              "σ̂k (t) is defined, this initialization also ensures proper posteriors [34].\n",
              "Top-two Thompson sampling is a reformulation of the Thompson sampling\n",
              "algorithm [54], such that it can be used in a pure-exploration context [51].\n",
              "Thompson sampling operates directly on the arms’ posterior of the mean µk .\n",
              "At each time step, Thompson sampling obtains one sample for each arm’s posterior. The arm with the highest sample is pulled, and its reward is subsequently\n",
              "used to update that arm’s posterior. While this approach has been proven\n",
              "highly successful to minimize cumulative regret [14, 3, 34], as it balances the\n",
              "exploration-exploitation trade-off, it is sub-optimal to identify the best arm [10].\n",
              "To adapt Thompson sampling to minimize simple regret, Top-two Thompson\n",
              "sampling increases the amount of exploration. To this end, an exploration probability ω needs to be specified. At each time step, one sample is obtained for\n",
              "each arm’s posterior. The arm Atop with the highest sample is only pulled with\n",
              "probability ω. With probability 1 − ω we repeat sampling from the posteriors\n",
              "until we find an arm Atop-2 that has the highest posterior sample and where\n",
              "Atop 6= Atop-2 . When the arm Atop-2 is found, it is pulled and the observed\n",
              "reward is used to update the posterior of the pulled arm. When the available\n",
              "budget is consumed, the arm with the highest average reward is recommended.\n",
              "As Top-two Thompson sampling only requires samples from the arms’ posteriors, we can use the t-distributed posteriors from Equation 5 as is. To avoid\n",
              "improper posteriors, each arm needs to be initialized 2 times [34].\n",
              "As specified in the previous subsection, the reward distribution is censored.\n",
              "We observe each reward, but only consider it to update the arm’s value when it\n",
              "exceeds the threshold T0 (i.e., when we receive a sample from the mode of the\n",
              "epidemic that represents the established epidemic).\n",
              "\n",
              "4.5\n",
              "\n",
              "Probability of success\n",
              "\n",
              "The probability that an arm recommendation is correct presents a useful confidence statistic to support policy makers with their decisions. As Top-two\n",
              "Thompson sampling recommends the arm with the highest average reward, the\n",
              "\n",
              "9\n",
              "\n",
              "\fprobability of success is\n",
              "P (µJ = max µk )\n",
              "1≤k≤K\n",
              "\n",
              "(15)\n",
              "\n",
              "where µJ is the random variable that represents the mean of the recommended\n",
              "arm.\n",
              "As we assume that the arm’s reward distributions are independent, this\n",
              "probability can be computed using the recommended arm’s posterior probability density function fµJ and the other arms’ cumulative density function Fµk :\n",
              "P (µJ = max µk ) = P (∩K\n",
              "k6=J (µk ≤ µJ ))\n",
              "1≤k≤K\n",
              "Z\n",
              "=\n",
              "P (∩K\n",
              "k6=J (µk ≤ x))P (µJ = x)dx\n",
              "x∈R\n",
              "\n",
              "=\n",
              "\n",
              "Z\n",
              "\n",
              "x∈R\n",
              "\n",
              "=\n",
              "\n",
              "Z\n",
              "\n",
              "x∈R\n",
              "\n",
              "K\n",
              "\u0002Y\n",
              "\n",
              "k6=J\n",
              "\n",
              "K\n",
              "\u0002Y\n",
              "\n",
              "k6=J\n",
              "\n",
              "\u0003\n",
              "P (µk ≤ x) P (µJ = x)dx\n",
              "\u0003\n",
              "Fµk (x) fµJ (x)dx\n",
              "\n",
              "As this integral cannot be computed analytically, we estimate it using Gaussian\n",
              "quadrature [2].\n",
              "It is important to notice that, while aiming for generality, we made some\n",
              "conservative assumptions: the reward distributions are approximated as Gaussian and the uninformative Jeffreys prior is used. These assumptions imply\n",
              "that the derived probability of success will be an under-estimator for the actual\n",
              "recommendation success.\n",
              "\n",
              "5\n",
              "\n",
              "Experiments\n",
              "\n",
              "We composed and performed an experiment in the context of pandemic influenza, where we analyze the mitigation strategy to vaccinate a population\n",
              "when only a limited number of vaccine doses is available (details about the rationale behind this scenario in Section 2.1). In our experiment, we extend the\n",
              "simulation environment presented in [39] to accommodate a realistic setting to\n",
              "evaluate vaccine allocation. In contrast to [39], we consider a large and realistic\n",
              "social network (i.e., the city of Seattle) and a wide range of R0 values.\n",
              "We consider the scenario when a pandemic is emerging in a particular geographical region and vaccines becomes available, albeit in a limited number of\n",
              "doses. When the number of vaccine doses is limited, it is imperative to identify an optimal vaccine allocation strategy [43]. In our experiment, we explore\n",
              "the allocation of vaccines over five different age groups: pre-school children,\n",
              "school-age children, young adults, older adults and the elderly as presented by\n",
              "Chao et al. [12]. We consider this experiment for a wide range of R0 values.\n",
              "\n",
              "10\n",
              "\n",
              "\f5.1\n",
              "\n",
              "Influenza model and configuration\n",
              "\n",
              "The epidemiological model used in the experiments is the FluTE stochastic\n",
              "individual-based model. In our experiment we consider the population of Seattle\n",
              "(United states) that includes 560,000 individuals [12]. This population is realistic both with respect to the number of individuals and its community structure,\n",
              "and provides an adequate setting for the validation of vaccine strategies [57].\n",
              "At the first day of the simulated epidemic, 10 random individuals are seeded\n",
              "with an infection. The epidemic is simulated for 180 days, during this time no\n",
              "more infections are seeded. Thus, all new infections established during the run\n",
              "time of the simulation, result from the mixing between infectious and susceptible\n",
              "individuals. We assume no pre-existing immunity towards the circulating virus\n",
              "variant. We choose the number of vaccine doses to allocate to be approximately\n",
              "4.5% of the population size [43].\n",
              "In this experiment, we explore the efficacy of different vaccine allocation\n",
              "strategies. We consider that only one vaccine variant is available in the simulation environment. FluTE allows vaccine efficacy to be configured on 3 levels: efficacy to protect against infection when an individual is susceptible (i.e.,\n",
              "V ESus ), efficacy to avoid an infected individual from becoming infectious (i.e.,\n",
              "V EInf ) and efficacy to avoid an infected individual from becoming symptomatic\n",
              "(i.e., V ESym ). In our experiment we choose V ESus = 0.5 [42], V EInf = 0.5 [42]\n",
              "and V ESym = 0.67 [59]. The influenza vaccine only becomes fully effective after\n",
              "a certain period upon its administration, and the effectiveness increases gradually over this period [1]. In our experiment, we assume the vaccine effectiveness\n",
              "to build up exponentially over a period of 2 weeks [1].\n",
              "We perform our experiment for a set of R0 values within the range of 1.4\n",
              "to 2.4, in steps of 0.2. This range is considered representative for the epidemic\n",
              "potential of influenza pandemics [8, 12, 43]. We refer to this set of R0 values as\n",
              "R0 .\n",
              "Note that the setting described in this subsection, in conjunction with a\n",
              "particular R0 value, corresponds to a model configuration (i.e., c0 ∈ C, Definition 2).\n",
              "The computational complexity of FluTE simulations depends both on the\n",
              "size of the susceptible population and the proportion of the population that\n",
              "becomes infected. For the population of Seattle, the simulation run time was up\n",
              "to 11 21 minutes (median of 10 21 minutes), on state-of-the-art hardware (details\n",
              "in Supplementary Information, section 7).\n",
              "\n",
              "5.2\n",
              "\n",
              "Formulating vaccine allocation strategies\n",
              "\n",
              "We consider 5 age groups to which vaccine doses can be allocated: pre-school\n",
              "children (i.e., 0-4 years old), school-age children (i.e., 5-18 years old), young\n",
              "adults (i.e., 19-29 years old), older adults (i.e., 30-64 years old) and the elderly\n",
              "(i.e., > 65 years old) [12]. An allocation scheme can be encoded as a Boolean 5tuple, where each position in the tuple corresponds to the respective age group.\n",
              "The Boolean value at a particular position in the tuple denotes whether vaccines\n",
              "\n",
              "11\n",
              "\n",
              "\fshould be allocated to the respective age group. When vaccine is to be allocated\n",
              "to a particular age group, this is done proportional to the size of the population\n",
              "that is part of this age group [43].\n",
              "To decide on the best vaccine allocation strategy, we enumerate all possible\n",
              "combinations of this tuple. The tuple can be encoded as a binary number, and\n",
              "as such the different allocation strategies can be represented by integers (i.e.,\n",
              "{0, ..., 31}).\n",
              "\n",
              "5.3\n",
              "\n",
              "An influenza preventive bandit\n",
              "\n",
              "The influenza preventive bandit BF lu has exactly 32 arms. Each arm Ak is\n",
              "associated with the allocation strategy for which the integer encoding is k.\n",
              "Given a model configuration c0 ∈ C (Definition 2), when an arm Ak of BF lu\n",
              "is pulled, FluTE is invoked with c0 and the vaccine allocation strategy pk ∈ P\n",
              "(Definition 2) associated with the arm Ak . When FluTE finishes, it outputs the\n",
              "proportion of the population that experienced a symptomatic infection pI , from\n",
              "which the reward rk = 1 − pI is computed.\n",
              "\n",
              "5.4\n",
              "\n",
              "Outcome distributions\n",
              "\n",
              "To establish a proxy for the ground truth concerning the outcome distributions\n",
              "of the 32 considered preventive strategies, all strategies were evaluated 1000\n",
              "times, for each of the R0 values in R0 . We will use this ground truth as a reference to validate the correctness of the recommendations obtained throughout\n",
              "our experiments.\n",
              "R0 presents us with an interesting evaluation problem. To demonstrate this,\n",
              "we visualize the outcome distribution for R0 = 1.4 in Figure 1 and for R0 = 2.4\n",
              "in Figure 2 (the outcome distributions for the other R0 values are shown in\n",
              "Section 3 of the Supplementary Information). Firstly, we observe that for different values of R0 , the distances between top arms’ means differ. Additionally,\n",
              "outcome distribution variances vary over the set of R0 values in R0 . These\n",
              "differences produce distinct levels of evaluation hardness (see Section 4.4), and\n",
              "demonstrate the setting’s usefulness as benchmark to evaluate preventive strategies. Secondly, we expect the outcome distribution to be bimodal, however, the\n",
              "probability to sample from the mode of the outcome distribution that represents\n",
              "the non-established epidemic decreases as R0 increases [40]. This expectation\n",
              "is confirmed when we inspect Figure 1, that shows a bimodal distribution for\n",
              "R0 = 1.4, while Figure 2 shows a unimodal outcome distribution for R0 = 2.4,\n",
              "as only samples from the established epidemic were obtained.\n",
              "Our analysis identified that the best vaccine allocation strategy was h0, 1, 0, 0, 0i\n",
              "(i.e., allocate vaccine to school children, strategy 8) for all R0 values in R0 .\n",
              "\n",
              "5.5\n",
              "\n",
              "Best-arm identification experiment\n",
              "\n",
              "To assess the performance of the different best-arm identification algorithms\n",
              "(i.e., Successive Rejects, BayesGap and Top-two Thompson sampling) we run\n",
              "12\n",
              "\n",
              "\f0.3\n",
              "\n",
              "epidemic size\n",
              "\n",
              "0.2\n",
              "\n",
              "0.1\n",
              "\n",
              "0\n",
              "1\n",
              "2\n",
              "3\n",
              "4\n",
              "5\n",
              "6\n",
              "7\n",
              "8\n",
              "9\n",
              "10\n",
              "11\n",
              "12\n",
              "13\n",
              "14\n",
              "15\n",
              "16\n",
              "17\n",
              "18\n",
              "19\n",
              "20\n",
              "21\n",
              "22\n",
              "23\n",
              "24\n",
              "25\n",
              "26\n",
              "27\n",
              "28\n",
              "29\n",
              "30\n",
              "31\n",
              "\n",
              "0.0\n",
              "\n",
              "vaccine allocation strategy\n",
              "Figure 1: Violin plot that depicts the density of the outcome distribution (i.e.,\n",
              "epidemic size) for 32 vaccine allocation strategies (Ro = 1.4).\n",
              "\n",
              "13\n",
              "\n",
              "\fepidemic size\n",
              "\n",
              "0.52\n",
              "\n",
              "0.51\n",
              "\n",
              "0.50\n",
              "\n",
              "0\n",
              "1\n",
              "2\n",
              "3\n",
              "4\n",
              "5\n",
              "6\n",
              "7\n",
              "8\n",
              "9\n",
              "10\n",
              "11\n",
              "12\n",
              "13\n",
              "14\n",
              "15\n",
              "16\n",
              "17\n",
              "18\n",
              "19\n",
              "20\n",
              "21\n",
              "22\n",
              "23\n",
              "24\n",
              "25\n",
              "26\n",
              "27\n",
              "28\n",
              "29\n",
              "30\n",
              "31\n",
              "\n",
              "0.49\n",
              "\n",
              "vaccine allocation strategy\n",
              "Figure 2: Violin plot that depicts the density of the outcome distribution (i.e.,\n",
              "epidemic size) for 32 vaccine allocation strategies (Ro = 2.4).\n",
              "\n",
              "14\n",
              "\n",
              "\feach algorithm for all budgets in the range of 32 to 500. This evaluation is\n",
              "performed on the influenza bandit game that we defined earlier. For each budget,\n",
              "we run the algorithms 100 times, and report the recommendation success rate.\n",
              "In the previous section, the optimal vaccine allocation strategy was identified\n",
              "to be h0, 1, 0, 0, 0i (i.e. vaccine allocation strategy 8) for all R0 in R0 . We thus\n",
              "consider a recommendation to be correct when it equals this vaccine allocation\n",
              "strategy.\n",
              "We evaluate the algorithm’s performance with respect to each other and with\n",
              "respect to uniform sampling, the current state-of-the art to evaluate preventive\n",
              "strategies. The uniform sampling method pulls arm Au for each step t of the\n",
              "given budget T , where Au ’s index u is sampled from the uniform distribution\n",
              "U(1, K). To consider different levels of hardness and obtain insight in the effect\n",
              "of the unestablished outcome distribution, we perform this analysis for each R0\n",
              "value in R0 .\n",
              "For the Bayesian best-arm identification algorithms, the prior specifications\n",
              "are detailed in Section 4.4. BayesGap requires an upper and lower bound that\n",
              "is defined in terms of the used posteriors. In our experiments, we use upper\n",
              "bound Uk (t) and lower bound Lk (t) that were established in Section 4.4. Toptwo Thompson sampling requires a parameter that modulates the amount of\n",
              "exploration ω. As it is important for best-arm identification algorithms to differentiate between the top two arms, we choose ω = 0.5, such that, in the limit,\n",
              "Top-two Thompson sampling will explore the top two arms uniformly.\n",
              "We censor the reward distribution based on the threshold T0 we defined\n",
              "in Section 4.3. This threshold depends on basic reproductive number R0 and\n",
              "dispersion parameter γ. R0 is defined explicitly for each of our experimental\n",
              "settings. For the dispersion parameter we choose γ = 0.5, which is a conservative\n",
              "choice according to the literature [16, 24]. We define the probability cutoff\n",
              "ℓ = 10−10 .\n",
              "Figure 3 and Figure 4 show recommendation success rate for each of the\n",
              "best-arm identification algorithms, respectively for R0 = 1.4 and R0 = 2.4. The\n",
              "results for the other R0 values are visualized in Section 4 of the Supplementary\n",
              "Information.\n",
              "The results for different values of R0 clearly indicate that our selection of\n",
              "best-arm identification algorithms significantly outperform the uniform sampling method. In our experiment, where we consider different R0 values, the\n",
              "uniform sampling method requires more than double the amount of evaluations\n",
              "to achieve a similar recommendation performance. For the harder problems\n",
              "(e.g., setting with R0 = 2.4), recommendation uncertainty remains considerable even after consuming 3 times the budget required by Top-two Thompson\n",
              "sampling.\n",
              "All best-arm identification algorithms require an initialization phase in order to output a well-defined recommendation. Successive Rejects needs to pull\n",
              "each arm at least once, while Top-two Thompson sampling and BayesGap need\n",
              "to pull each arm respectively 2 and 3 times (details in Section 4.4). For this\n",
              "reason, these algorithms’ performance can only be evaluated after this initialization phase. BayesGap’s performance is on par with Successive Rejects, except\n",
              "15\n",
              "\n",
              "\ffor the hardest setting we studied (i.e., R0 = 2.4). In comparison, Top-two\n",
              "Thompson sampling consistently outperforms Successive Rejects 30 pulls after\n",
              "the initialization phase.\n",
              "Top-two Thompson sampling needs to initialize each arm’s posterior with\n",
              "2 pulls, i.e., double the amount of uniform sampling and Successive Rejects.\n",
              "However, our experiments clearly show that none of the other algorithms reach\n",
              "any acceptable recommendation rate using less than 64 pulls, thereby alleviating\n",
              "concerns using a t-distributed posterior.\n",
              "\n",
              "1.0\n",
              "\n",
              "success rate\n",
              "\n",
              "0.8\n",
              "0.6\n",
              "0.4\n",
              "BG\n",
              "SR\n",
              "TtTs\n",
              "Uni\n",
              "\n",
              "0.2\n",
              "0.0\n",
              "100\n",
              "\n",
              "200\n",
              "\n",
              "300\n",
              "budget\n",
              "\n",
              "400\n",
              "\n",
              "500\n",
              "\n",
              "Figure 3: In this figure, we present the results for the experiment with R0 =\n",
              "1.4. Each curve represents the rate of successful arm recommendations (y-axis)\n",
              "for a range of budgets (x-axis). A curve is shown for each of the considered\n",
              "algorithms: BayesGap (legend: BG), Successive Rejects (legend: SR), Top-two\n",
              "Thompson sampling (legend: TtTs) and Uniform sampling (legend: Uni).\n",
              "In Section 4 we derived a statistic to express the probability of success (Ps )\n",
              "concerning a recommendation made by Top-two Thompson sampling. We analyzed this probability for all the Top-two Thompson sampling recommendations\n",
              "that were obtained in the experiment described above. To provide some insights\n",
              "on how this statistic can be used to support policy makers, we show the Ps values\n",
              "of all Top-two Thompson sampling recommendations for R0 = 2.4 in Figure 5\n",
              "(Figures for the other R0 values in Section 5 of the Supplementary Information).\n",
              "Figure 5 indicates that Ps closely follows recommendation correctness and that\n",
              "\n",
              "16\n",
              "\n",
              "\f1.0\n",
              "\n",
              "success rate\n",
              "\n",
              "0.8\n",
              "0.6\n",
              "0.4\n",
              "BG\n",
              "SR\n",
              "TtTs\n",
              "Uni\n",
              "\n",
              "0.2\n",
              "0.0\n",
              "100\n",
              "\n",
              "200\n",
              "\n",
              "300\n",
              "budget\n",
              "\n",
              "400\n",
              "\n",
              "500\n",
              "\n",
              "Figure 4: In this figure, we present the results for the experiment with R0 =\n",
              "2.4. Each curve represents the rate of successful arm recommendations (y-axis)\n",
              "for a range of budgets (x-axis). A curve is shown for each of the considered\n",
              "algorithms: BayesGap (legend: BG), Successive Rejects (legend: SR), Top-two\n",
              "Thompson sampling (legend: TtTs) and Uniform sampling (legend: Uni).\n",
              "the uncertainty of Ps is inversely proportional to the size of the available budget.\n",
              "Additionally, in Figure 6 (Figures for the other R0 values in Section 6 of the\n",
              "Supplementary Information) we confirm that Ps underestimates recommendation correctness. These observations indicate that Ps has the potential to serve\n",
              "as a conservative statistic to inform policy makers about the confidence of a particular recommendation, and thus can be used to define meaningful cutoffs to\n",
              "guide policy makers in their interpretation of the recommendation of preventive\n",
              "strategies.\n",
              "\n",
              "6\n",
              "\n",
              "Conclusion\n",
              "\n",
              "We formulate the objective to select the best preventive strategy in an individualbased model as a fixed budget best-arm identification problem. An experiment\n",
              "was set up to evaluate this setting in the context of pandemic influenza. To\n",
              "assess the best arm recommendation performance of the preventive bandit, we\n",
              "report a success rate over 100 independent bandit runs.\n",
              "17\n",
              "\n",
              "\f1.0\n",
              "probability of success\n",
              "\n",
              "0.9\n",
              "0.8\n",
              "0.7\n",
              "0.6\n",
              "0.5\n",
              "0.4\n",
              "\n",
              "success\n",
              "failure\n",
              "\n",
              "0.3\n",
              "100\n",
              "\n",
              "200\n",
              "\n",
              "300\n",
              "budget\n",
              "\n",
              "400\n",
              "\n",
              "500\n",
              "\n",
              "Figure 5: Top-two Thompson sampling was run 100 times for each budget for the\n",
              "experiment with R0 = 2.4. For each of the recommendations, Ps was computed.\n",
              "These Ps values are shown as a scatter plot, where each point’s color reflects\n",
              "the correctness of the recommendation (see legend).\n",
              "We demonstrate that it is possible to efficiently identify the optimal preventive strategy using only a limited number of model evaluations, even if there\n",
              "is a large number of preventive strategies to consider. Compared to uniform\n",
              "sampling, our method is able to recommend the best preventive strategy reducing the number of required model evaluations 2-to-3 times. Additionally, we\n",
              "show that by using Bayesian best-arm identification algorithms, statistics can\n",
              "be defined to support policy makers with their decisions. As such, we are confident that our method has the potential to be used as a decision support tool\n",
              "for mitigating epidemics. This will enable the use of individual-based models in\n",
              "studies where it would otherwise be computationally too prohibitive, and allow\n",
              "researchers to explore a wider variety of model scenarios.\n",
              "We identify two particular directions for future work. Firstly, while our\n",
              "method is evaluated in the context of pandemic influenza, it is important to\n",
              "stress that it can be used to evaluate preventive strategies for other infectious\n",
              "diseases. Since recently, a Dengue vaccine is available [28], and the optimal allocation of this vaccine remains an important research topic [23, 4], we recognize\n",
              "that Dengue epidemics are an interesting use case. Secondly, in this paper, our\n",
              "\n",
              "18\n",
              "\n",
              "\fempirical success rate\n",
              "\n",
              "1.0\n",
              "0.9\n",
              "0.8\n",
              "0.7\n",
              "0.6\n",
              "0.5\n",
              "0.5\n",
              "\n",
              "0.6\n",
              "0.7\n",
              "0.8\n",
              "0.9\n",
              "estimated probability of succes\n",
              "\n",
              "1.0\n",
              "\n",
              "Figure 6: Top-two Thompson sampling was run 100 times for each budget for the\n",
              "experiment with R0 = 2.4. For each of the recommendations, Ps was computed.\n",
              "The Ps values were binned: 0.5 to 1 in steps of 0.05. Per bin, we thus have a set\n",
              "of Bernoulli trials, for which we show the empirical success rate (blue scatter)\n",
              "and the Clopper-Pearson confidence interval (blue confidence bounds) [15]. The\n",
              "orange reference line denotes perfect correlation between the empirical success\n",
              "rate and the estimated probability of success.\n",
              "preventive bandits only learn with respect to a single model outcome (i.e., the\n",
              "proportion of symptomatic infections). However, for many pathogens it is interesting to incorporate multiple objectives (e.g., morbidity, mortality, cost). In\n",
              "the future, we aim to use multi-objective multi-armed bandits [17] in contrast to\n",
              "the current single-objective preventive bandits. With this approach, we plan to\n",
              "learn a coverage set containing an optimal strategy for every possible preference\n",
              "profile the decision makers might have [50].\n",
              "Statement with respect to the reproducibility of our research: if this manuscript\n",
              "is accepted, all source code used in our experiments will be made publicly available.\n",
              "\n",
              "19\n",
              "\n",
              "\fAcknowledgments\n",
              "Pieter Libin was supported by a PhD grant of the FWO (Fonds Wetenschappelijk Onderzoek Vlaanderen) and the VUB research council (VUB/OZR2714).\n",
              "Timothy Verstraeten was supported by a PhD grant of the FWO (Fonds Wetenschappelijk Onderzoek Vlaanderen) and the VUB research council (VUB/OZR2884).\n",
              "Kristof Theys was supported by a postdoctoral grant of the FWO (Fonds Wetenschappelijk Onderzoek Vlaanderen). Diederik Roijers was supported by a postdoctoral grant of the FWO (Fonds Wetenschappelijk Onderzoek Vlaanderen).\n",
              "The computational resources and services used in this work were provided by\n",
              "the Hercules Foundation and the Flemish Government department EWI-FWO\n",
              "Krediet aan Navorsers (Theys, KAN2012 1.5.249.12.).\n",
              "\n",
              "20\n",
              "\n",
              "\fSupplementary Information\n",
              "\n",
              "S1\n",
              "\n",
              "Introduction\n",
              "\n",
              "In this Supplementary Information we provide a proof for BayesGap’s simple\n",
              "regret bound (Section 2). Furthermore, we provide additional figures that were\n",
              "omitted from the main manuscript: figures for the outcome (i.e., epidemic size)\n",
              "distributions (Section 3), figures for the experimental success rates (Section 4),\n",
              "figures for the probabilities of success (i.e., Ps values) per budget (Section 5)\n",
              "and figures for the binned distribution over Ps values (Section 6). Finally, in\n",
              "Section 7, we describe the computational resources that were used to execute\n",
              "the simulations.\n",
              "\n",
              "S2\n",
              "\n",
              "BayesGap simple regret bound for T-distributed\n",
              "posteriors\n",
              "\n",
              "Lemma 1. Consider a Jeffrey’s prior (µk , σk2 ) ∼ σk−3 over the parameters of\n",
              "the Gaussian reward distributions. Then the posterior mean of arm k has the\n",
              "following nonstandardized t-distribution at pull nk :\n",
              "p\n",
              "µk | x̄k,nk , Sk,nk ∼ Tnk (x̄k,nk , n−1\n",
              "Sk,nk )\n",
              "k\n",
              "\n",
              "where nk is the number of pulls for arm k, x̄k,nk is the sample mean and Sk,nk\n",
              "is the sum of squares.\n",
              "Proof. This lemma was presented and proved by Honda et al. [34].\n",
              "\n",
              "Lemma 2. Consider a random variable X ∼ Tν (µ, λ) with variance σ 2 =\n",
              "ν\n",
              "2\n",
              "ν−2 λ , ν > 2 and β > 0. The probability that X is within a radius βσ from its\n",
              "mean can then be written as:\n",
              "P (|X − µ| < βσ) ≥ 1 − 2\n",
              "where\n",
              "C(ν) =\n",
              "\n",
              "ν C(ν)\n",
              "ν−1 β\n",
              "\n",
              "\u0012\n",
              "\u0013−0.5(ν−1)\n",
              "β2\n",
              "1+\n",
              "ν\n",
              "\n",
              "Γ(0.5ν + 0.5)\n",
              "√\n",
              "Γ(0.5ν) πν\n",
              "\n",
              "is the normalizing constant of a standard t-distribution.\n",
              "Proof. Consider a random variable Z ∼ Tν (0, 1), ν > 2 and β > 0. Then the\n",
              "\n",
              "1\n",
              "\n",
              "\fprobability of Z being greater than β\n",
              "P (Z > β\n",
              "\n",
              "r\n",
              "\n",
              "Z\n",
              "\n",
              "ν\n",
              "(1)\n",
              ")=\n",
              "ν−2\n",
              "\n",
              "+∞\n",
              "\n",
              "β\n",
              "\n",
              "√\n",
              "\n",
              "ν\n",
              "ν−2\n",
              "\n",
              "q\n",
              "\n",
              "ν\n",
              "ν−2\n",
              "\n",
              "is:\n",
              "\n",
              "Tν (z | 0, 1)dz\n",
              "\n",
              "\u0012\n",
              "\u0013−0.5(ν+1)\n",
              "z2\n",
              "1\n",
              "+\n",
              "dz\n",
              "√ ν\n",
              "ν\n",
              "β ν−2\n",
              "\u0012\n",
              "\u0013−0.5(ν+1)\n",
              "Z +∞\n",
              "(2)\n",
              "z2\n",
              "z\n",
              "q\n",
              "1+\n",
              "dz\n",
              "≤ C(ν) √\n",
              "ν\n",
              "ν\n",
              "ν\n",
              "β ν−2\n",
              "β ν−2\n",
              "√\n",
              "\u0013−0.5(ν+1)\n",
              "\u0012\n",
              "Z +∞\n",
              "ν−2\n",
              "ν\n",
              "ν −1\n",
              "z2\n",
              "√ C(ν) √\n",
              "=−\n",
              "−\n",
              "z 1+\n",
              "dz\n",
              "ν\n",
              "ν−1 β ν\n",
              "ν\n",
              "ν\n",
              "β ν−2\n",
              "p\n",
              "\u0012\n",
              "\u0013−0.5(ν−1) +∞\n",
              "ν(ν − 2) C(ν)\n",
              "z2\n",
              "(3)\n",
              "1+\n",
              "= −\n",
              "ν−1\n",
              "β\n",
              "ν\n",
              "√ ν\n",
              "\n",
              "= C(ν)\n",
              "\n",
              "Z\n",
              "\n",
              "+∞\n",
              "\n",
              "β\n",
              "\n",
              "ν−2\n",
              "\n",
              "p\n",
              "\u0012\n",
              "\u0013−0.5(ν−1)\n",
              "ν(ν − 2) C(ν)\n",
              "β2\n",
              "(4)\n",
              "1+\n",
              "=\n",
              "ν −1\n",
              "β\n",
              "ν −2\n",
              "q\n",
              "ν\n",
              "The probability of Z being greater than the lower bound β ν−2\n",
              "is the integral\n",
              "\n",
              "over its probability density function, starting from that lower bound (1). In the\n",
              "integral, we introduce a factor β √z ν , which is greater than 1 for the considered\n",
              "ν−2\n",
              "\n",
              "values of z (2). We then take note of the following derivative, and use this result\n",
              "to analytically solve the integral (3):\n",
              "d\n",
              "dx\n",
              "\n",
              "\u0012\n",
              "\u0013−0.5(ν−1)\n",
              "\u0013−0.5(ν+1)\n",
              "\u0012\n",
              "x2\n",
              "x2\n",
              "ν −1\n",
              "1+\n",
              "x 1+\n",
              "=−\n",
              "ν\n",
              "ν\n",
              "ν\n",
              "\n",
              "Finally, we solve the primitive from β √z ν\n",
              "\n",
              "to infinity (4).\n",
              "\n",
              "ν−2\n",
              "\n",
              "Next, we apply a union bound to obtain\n",
              "q a lower bound on the probability\n",
              "ν\n",
              "that the magnitude of Z is smaller than β ν−2\n",
              ":\n",
              "P (|Z| < β\n",
              "\n",
              "r\n",
              "\n",
              "ν\n",
              ")≥1−2\n",
              "ν −2\n",
              "\n",
              "Finally, consider Z =\n",
              "P (|X − µ| < β\n",
              "\n",
              "r\n",
              "\n",
              "p\n",
              "\u0012\n",
              "\u0013−0.5(ν−1)\n",
              "ν(ν − 2) C(ν)\n",
              "β2\n",
              "1+\n",
              "ν −1\n",
              "β\n",
              "ν −2\n",
              "\n",
              "(X−µ)\n",
              ":\n",
              "λ\n",
              "\n",
              "ν\n",
              "λ) ≥ 1 − 2\n",
              "ν −2\n",
              "\n",
              "p\n",
              "\u0012\n",
              "\u0013−0.5(ν−1)\n",
              "ν(ν − 2) C(ν)\n",
              "β2\n",
              "1+\n",
              "ν −1\n",
              "β\n",
              "ν −2\n",
              "\n",
              "Lemma 3. Consider a K-armed bandit problem with budget T and K arms.\n",
              "Let Uk (t) and Lk (t) be upper and lower bounds that hold for all times t ≤ T and\n",
              "2\n",
              "\n",
              "\fall arms k ≤ K with probability 1 − δk (t). Finally, let gk be a monotonically\n",
              "K\n",
              "P\n",
              "gk−1 (Hk,ǫ ) ≤\n",
              "decreasing function such that Uk (t) − Lk (t) ≤ gk (nk (t − 1)) and\n",
              "\n",
              "T − K. We can then bound the simple regret RT as:\n",
              "P (RT < ǫ) ≥ 1 −\n",
              "\n",
              "K X\n",
              "T\n",
              "X\n",
              "\n",
              "k=1\n",
              "\n",
              "δk (t)\n",
              "\n",
              "k=1 t=1\n",
              "\n",
              "Proof. First, we define E as the event in which every mean µk is bounded by its\n",
              "associated bounds (i.e., Uk (t) and Lk (t)) for each time step [33].\n",
              "E := ∀k ≤ K, ∀t ≤ T : Lk (t) ≤ µk ≤ Uk (t)\n",
              "The probability of µk deviating from a single bound at time t is by definition\n",
              "T\n",
              "K P\n",
              "P\n",
              "δk (t).\n",
              "δk (t). When applying the union bound, we obtain P (E) ≥ 1 −\n",
              "k=1 t=1\n",
              "\n",
              "The probability of regret is equal to the probability of the event E occuring, as\n",
              "proven in [33].\n",
              "Theorem 1. Consider a K-armed Gaussian bandit problem with budget T and\n",
              "2\n",
              "unknown variance. Let σG\n",
              "be a generalization of that variance over all arms,\n",
              "and Uk (t) and Lk (t) respectively be the upper and lower bounds for each arm k\n",
              "at time t, where Uk (t) = µ̂k (t) + β σ̂k (t) and Lk (t) = µ̂k (t) − β σ̂k (t). The simple\n",
              "regret is then bounded as:\n",
              "p\n",
              "\u0012\n",
              "\u0013−0.5(nk (t)−1)\n",
              "nk (t)(nk (t) − 2) C(nk (t))\n",
              "β2\n",
              "P (RT ≤ ǫ) ≥ 1 − 2\n",
              "1+\n",
              "nk (t) − 1\n",
              "β\n",
              "nk (t) − 2\n",
              "k=1 t=1\n",
              "\n",
              "\n",
              "\n",
              "−0.5 min nk (t)\n",
              "k,t\n",
              "2\n",
              "β\n",
              "\n",
              "\n",
              "\n",
              "\n",
              "≥ 1 − O KT 1 +\n",
              "\n",
              "min nk (t)\n",
              "K X\n",
              "T\n",
              "X\n",
              "\n",
              "k,t\n",
              "\n",
              "where:\n",
              "\n",
              "β=\n",
              "\n",
              "s\n",
              "\n",
              "T − 3K\n",
              "2\n",
              "4Hǫ σG\n",
              "\n",
              "Note that when min nk (t) → +∞, the bound decreases exponentially in β, simik,t\n",
              "\n",
              "lar to the problem setting presented in [33]. Intuitively, this result makes sense,\n",
              "as for known variances, a Gaussian can be used to describe the posterior means,\n",
              "and indeed, as the number of pulls approaches infinity, our t-distributions converge to Gaussians.\n",
              "Proof. According to Lemma 1, the posterior over the average reward is a t-\n",
              "\n",
              "3\n",
              "\n",
              "\fdistribution with scaling factor λk (t) = nk (t)−1\n",
              "\n",
              "p\n",
              "Sk,nk (t) . Therefore,\n",
              "\n",
              "Uk (t + 1) − Lk (t + 1) = 2β σ̂k (t)\n",
              "p\n",
              "(1)\n",
              "= 2β nk (t)(nk (t) − 2)−1 λk (t)2\n",
              "q\n",
              "(2)\n",
              "= nk (t)(nk (t) − 2)−1 nk (t)−2 Sk,nk (t)\n",
              "s\n",
              "Sk,nk (t)\n",
              "= (nk (t) − 2)−1\n",
              "nk (t)\n",
              "q\n",
              "(3)\n",
              "= (nk (t) − 2)−1 s2k (t)\n",
              "(4)\n",
              "\n",
              "= gk (nk (t))\n",
              "\n",
              "k (t)\n",
              "λk (t)2 for arm k at time t, with\n",
              "The variance of a t-distribution equals nkn(t)−2\n",
              "scaling factor λk (t) as described in Lemma 1 (1 + 2). We denote the variance\n",
              "over rewards per arm as s2k (t) (3) and define gk (nk (t)) to be the upper bound\n",
              "expression as specified in Lemma 3 (4).\n",
              "Next, we compute the inverse of gk (n):\n",
              "\n",
              "gk−1 (m) =\n",
              "\n",
              "4β 2 s2k (t)\n",
              "+2\n",
              "m2\n",
              "\n",
              "2\n",
              "We generalize s2k (t) to a variance σG\n",
              "representative\n",
              "for all arms.2 Approximating\n",
              "P\n",
              "−2\n",
              "the hardness of the problem as Hǫ = k Hk,ǫ , where Hk,ǫ is the arm-dependent\n",
              "hardness defined in [33], we obtain β as follows:\n",
              "K\n",
              "X\n",
              "\n",
              "k=1\n",
              "\n",
              "2\n",
              "gk−1 (Hkǫ ) ≈ 4β 2 σG\n",
              "Hǫ + 2K = T − K\n",
              "\n",
              "⇔β=\n",
              "\n",
              "s\n",
              "\n",
              "T − 3K\n",
              "2\n",
              "4Hǫ σG\n",
              "\n",
              "Finally, as the conditions in Lemma 3 on the function gk are now satisfied,\n",
              "the simple regret bound can be obtained using Lemma 3 and the probability\n",
              "that the true mean is out of the arm-specific bounds Uk (t) and Lk (t), given in\n",
              "Lemma 2.\n",
              "\n",
              "2 In the main paper, we choose σ 2 = s̄2 to be the mean over all arm-specific variances\n",
              "G\n",
              "G\n",
              "obtained after the initialization phase.\n",
              "\n",
              "4\n",
              "\n",
              "\fS3\n",
              "\n",
              "vaccine allocation strategy\n",
              "\n",
              "vaccine allocation strategy\n",
              "\n",
              "0.0\n",
              "\n",
              "0.1\n",
              "\n",
              "0.2\n",
              "\n",
              "0.3\n",
              "\n",
              "Outcome (i.e., epidemic size) distributions\n",
              "0.3\n",
              "\n",
              "0.2\n",
              "\n",
              "0.1\n",
              "\n",
              "0.0\n",
              "\n",
              "(b) Outcome distributions for R0 = 1.6.\n",
              "\n",
              "0.4\n",
              "\n",
              "0.3\n",
              "\n",
              "0.2\n",
              "\n",
              "0.1\n",
              "\n",
              "vaccine allocation strategy\n",
              "\n",
              "0.0\n",
              "\n",
              "0.50\n",
              "\n",
              "0.51\n",
              "\n",
              "0.52\n",
              "\n",
              "(d) Outcome distributions for R0 = 2.0.\n",
              "\n",
              "vaccine allocation strategy\n",
              "\n",
              "(a) Outcome distributions for R0 = 1.4.\n",
              "\n",
              "0.4\n",
              "\n",
              "0.3\n",
              "\n",
              "0.2\n",
              "\n",
              "0.1\n",
              "\n",
              "0.0\n",
              "\n",
              "0.5\n",
              "\n",
              "(c) Outcome distributions for R0 = 1.8.\n",
              "\n",
              "0.4\n",
              "\n",
              "0.3\n",
              "\n",
              "0.2\n",
              "\n",
              "0.49\n",
              "\n",
              "(f) Outcome distributions for R0 = 2.4.\n",
              "\n",
              "vaccine allocation strategy\n",
              "\n",
              "0\n",
              "1\n",
              "2\n",
              "3\n",
              "4\n",
              "5\n",
              "6\n",
              "7\n",
              "8\n",
              "9\n",
              "10\n",
              "11\n",
              "12\n",
              "13\n",
              "14\n",
              "15\n",
              "16\n",
              "17\n",
              "18\n",
              "19\n",
              "20\n",
              "21\n",
              "22\n",
              "23\n",
              "24\n",
              "25\n",
              "26\n",
              "27\n",
              "28\n",
              "29\n",
              "30\n",
              "31\n",
              "\n",
              "0.1\n",
              "\n",
              "vaccine allocation strategy\n",
              "\n",
              "(e) Outcome distributions for R0 = 2.2.\n",
              "\n",
              "0.0\n",
              "\n",
              "5\n",
              "\n",
              "0\n",
              "1\n",
              "2\n",
              "3\n",
              "4\n",
              "5\n",
              "6\n",
              "7\n",
              "8\n",
              "9\n",
              "10\n",
              "11\n",
              "12\n",
              "13\n",
              "14\n",
              "15\n",
              "16\n",
              "17\n",
              "18\n",
              "19\n",
              "20\n",
              "21\n",
              "22\n",
              "23\n",
              "24\n",
              "25\n",
              "26\n",
              "27\n",
              "28\n",
              "29\n",
              "30\n",
              "31\n",
              "\n",
              "epidemic size\n",
              "\n",
              "0\n",
              "1\n",
              "2\n",
              "3\n",
              "4\n",
              "5\n",
              "6\n",
              "7\n",
              "8\n",
              "9\n",
              "10\n",
              "11\n",
              "12\n",
              "13\n",
              "14\n",
              "15\n",
              "16\n",
              "17\n",
              "18\n",
              "19\n",
              "20\n",
              "21\n",
              "22\n",
              "23\n",
              "24\n",
              "25\n",
              "26\n",
              "27\n",
              "28\n",
              "29\n",
              "30\n",
              "31\n",
              "\n",
              "epidemic size\n",
              "\n",
              "epidemic size\n",
              "epidemic size\n",
              "epidemic size\n",
              "\n",
              "epidemic size\n",
              "0\n",
              "1\n",
              "2\n",
              "3\n",
              "4\n",
              "5\n",
              "6\n",
              "7\n",
              "8\n",
              "9\n",
              "10\n",
              "11\n",
              "12\n",
              "13\n",
              "14\n",
              "15\n",
              "16\n",
              "17\n",
              "18\n",
              "19\n",
              "20\n",
              "21\n",
              "22\n",
              "23\n",
              "24\n",
              "25\n",
              "26\n",
              "27\n",
              "28\n",
              "29\n",
              "30\n",
              "31\n",
              "0\n",
              "1\n",
              "2\n",
              "3\n",
              "4\n",
              "5\n",
              "6\n",
              "7\n",
              "8\n",
              "9\n",
              "10\n",
              "11\n",
              "12\n",
              "13\n",
              "14\n",
              "15\n",
              "16\n",
              "17\n",
              "18\n",
              "19\n",
              "20\n",
              "21\n",
              "22\n",
              "23\n",
              "24\n",
              "25\n",
              "26\n",
              "27\n",
              "28\n",
              "29\n",
              "30\n",
              "31\n",
              "0\n",
              "1\n",
              "2\n",
              "3\n",
              "4\n",
              "5\n",
              "6\n",
              "7\n",
              "8\n",
              "9\n",
              "10\n",
              "11\n",
              "12\n",
              "13\n",
              "14\n",
              "15\n",
              "16\n",
              "17\n",
              "18\n",
              "19\n",
              "20\n",
              "21\n",
              "22\n",
              "23\n",
              "24\n",
              "25\n",
              "26\n",
              "27\n",
              "28\n",
              "29\n",
              "30\n",
              "31\n",
              "\n",
              "\fBandit run success rates\n",
              "1.0\n",
              "\n",
              "1.0\n",
              "\n",
              "0.8\n",
              "\n",
              "0.8\n",
              "success rate\n",
              "\n",
              "success rate\n",
              "\n",
              "S4\n",
              "\n",
              "0.6\n",
              "0.4\n",
              "BG\n",
              "SR\n",
              "TtTs\n",
              "Uni\n",
              "\n",
              "0.2\n",
              "0.0\n",
              "100\n",
              "\n",
              "200\n",
              "\n",
              "300\n",
              "budget\n",
              "\n",
              "400\n",
              "\n",
              "100\n",
              "\n",
              "1.0\n",
              "\n",
              "0.8\n",
              "\n",
              "0.8\n",
              "\n",
              "0.6\n",
              "0.4\n",
              "BG\n",
              "SR\n",
              "TtTs\n",
              "Uni\n",
              "\n",
              "0.0\n",
              "100\n",
              "\n",
              "200\n",
              "\n",
              "300\n",
              "budget\n",
              "\n",
              "400\n",
              "\n",
              "100\n",
              "\n",
              "success rate\n",
              "\n",
              "0.8\n",
              "\n",
              "0.4\n",
              "BG\n",
              "SR\n",
              "TtTs\n",
              "Uni\n",
              "400\n",
              "\n",
              "200\n",
              "\n",
              "300\n",
              "budget\n",
              "\n",
              "400\n",
              "\n",
              "500\n",
              "\n",
              "(d) Bandit run results for R0 = 2.0.\n",
              "\n",
              "0.6\n",
              "\n",
              "300\n",
              "budget\n",
              "\n",
              "BG\n",
              "SR\n",
              "TtTs\n",
              "Uni\n",
              "\n",
              "0.0\n",
              "\n",
              "500\n",
              "\n",
              "0.8\n",
              "\n",
              "200\n",
              "\n",
              "500\n",
              "\n",
              "0.4\n",
              "\n",
              "1.0\n",
              "\n",
              "100\n",
              "\n",
              "400\n",
              "\n",
              "0.6\n",
              "\n",
              "1.0\n",
              "\n",
              "0.0\n",
              "\n",
              "300\n",
              "budget\n",
              "\n",
              "0.2\n",
              "\n",
              "(c) Bandit run results for R0 = 1.8.\n",
              "\n",
              "0.2\n",
              "\n",
              "200\n",
              "\n",
              "(b) Bandit run results for R0 = 1.6.\n",
              "\n",
              "1.0\n",
              "\n",
              "0.2\n",
              "\n",
              "BG\n",
              "SR\n",
              "TtTs\n",
              "Uni\n",
              "\n",
              "0.0\n",
              "\n",
              "500\n",
              "\n",
              "success rate\n",
              "\n",
              "success rate\n",
              "\n",
              "0.4\n",
              "0.2\n",
              "\n",
              "(a) Bandit run results for R0 = 1.4.\n",
              "\n",
              "success rate\n",
              "\n",
              "0.6\n",
              "\n",
              "0.6\n",
              "0.4\n",
              "BG\n",
              "SR\n",
              "TtTs\n",
              "Uni\n",
              "\n",
              "0.2\n",
              "0.0\n",
              "\n",
              "500\n",
              "\n",
              "100\n",
              "\n",
              "(e) Bandit run results for R0 = 2.2.\n",
              "\n",
              "200\n",
              "\n",
              "300\n",
              "budget\n",
              "\n",
              "400\n",
              "\n",
              "500\n",
              "\n",
              "(f) Bandit run results for R0 = 2.4.\n",
              "\n",
              "6\n",
              "\n",
              "\fS5\n",
              "\n",
              "Ps values for Top-two Thompson sampling\n",
              "1.0\n",
              "\n",
              "0.9\n",
              "\n",
              "probability of success\n",
              "\n",
              "probability of success\n",
              "\n",
              "1.0\n",
              "\n",
              "0.8\n",
              "0.7\n",
              "0.6\n",
              "0.5\n",
              "\n",
              "success\n",
              "failure\n",
              "\n",
              "0.4\n",
              "100\n",
              "\n",
              "200\n",
              "\n",
              "300\n",
              "budget\n",
              "\n",
              "400\n",
              "\n",
              "0.9\n",
              "0.8\n",
              "0.7\n",
              "\n",
              "500\n",
              "\n",
              "100\n",
              "\n",
              "1.0\n",
              "\n",
              "1.0\n",
              "\n",
              "0.9\n",
              "\n",
              "0.9\n",
              "\n",
              "0.8\n",
              "0.7\n",
              "\n",
              "0.5\n",
              "\n",
              "success\n",
              "failure\n",
              "100\n",
              "\n",
              "200\n",
              "\n",
              "300\n",
              "budget\n",
              "\n",
              "400\n",
              "\n",
              "500\n",
              "\n",
              "100\n",
              "\n",
              "0.9\n",
              "probability of success\n",
              "\n",
              "probability of success\n",
              "\n",
              "200\n",
              "\n",
              "300\n",
              "budget\n",
              "\n",
              "400\n",
              "\n",
              "500\n",
              "\n",
              "(d) Ps values for R0 = 2.0.\n",
              "\n",
              "0.8\n",
              "0.7\n",
              "0.6\n",
              "0.5\n",
              "\n",
              "0.8\n",
              "0.7\n",
              "0.6\n",
              "0.5\n",
              "0.4\n",
              "\n",
              "success\n",
              "failure\n",
              "400\n",
              "\n",
              "success\n",
              "failure\n",
              "\n",
              "0.5\n",
              "\n",
              "1.0\n",
              "\n",
              "300\n",
              "budget\n",
              "\n",
              "500\n",
              "\n",
              "0.6\n",
              "\n",
              "0.9\n",
              "\n",
              "200\n",
              "\n",
              "400\n",
              "\n",
              "0.7\n",
              "\n",
              "1.0\n",
              "\n",
              "100\n",
              "\n",
              "300\n",
              "budget\n",
              "\n",
              "0.8\n",
              "\n",
              "(c) Ps values for R0 = 1.8.\n",
              "\n",
              "0.4\n",
              "\n",
              "200\n",
              "\n",
              "(b) Ps values for R0 = 1.6.\n",
              "\n",
              "probability of success\n",
              "\n",
              "probability of success\n",
              "\n",
              "(a) Ps values for R0 = 1.4.\n",
              "\n",
              "0.6\n",
              "\n",
              "success\n",
              "failure\n",
              "\n",
              "0.6\n",
              "\n",
              "success\n",
              "failure\n",
              "\n",
              "0.3\n",
              "\n",
              "500\n",
              "\n",
              "100\n",
              "\n",
              "(e) Ps values for R0 = 2.2.\n",
              "\n",
              "200\n",
              "\n",
              "300\n",
              "budget\n",
              "\n",
              "400\n",
              "\n",
              "500\n",
              "\n",
              "(f) Ps values for R0 = 2.4.\n",
              "\n",
              "7\n",
              "\n",
              "\fS6\n",
              "\n",
              "Binned distribution of Ps values for Top-two\n",
              "Thompson sampling\n",
              "1.0\n",
              "\n",
              "0.9\n",
              "\n",
              "empirical success rate\n",
              "\n",
              "empirical success rate\n",
              "\n",
              "1.0\n",
              "\n",
              "0.8\n",
              "0.7\n",
              "0.6\n",
              "0.5\n",
              "0.4\n",
              "0.5\n",
              "\n",
              "0.6\n",
              "0.7\n",
              "0.8\n",
              "0.9\n",
              "estimated probability of succes\n",
              "\n",
              "0.4\n",
              "0.2\n",
              "\n",
              "0.5\n",
              "\n",
              "0.6\n",
              "0.7\n",
              "0.8\n",
              "0.9\n",
              "estimated probability of succes\n",
              "\n",
              "1.0\n",
              "\n",
              "(b) Binned distribution for R0 = 1.6.\n",
              "\n",
              "1.0\n",
              "\n",
              "1.0\n",
              "\n",
              "0.8\n",
              "\n",
              "empirical success rate\n",
              "\n",
              "empirical success rate\n",
              "\n",
              "0.6\n",
              "\n",
              "0.0\n",
              "\n",
              "1.0\n",
              "\n",
              "(a) Binned distribution for R0 = 1.4.\n",
              "\n",
              "0.6\n",
              "0.4\n",
              "0.2\n",
              "\n",
              "0.8\n",
              "0.6\n",
              "0.4\n",
              "0.2\n",
              "\n",
              "0.0\n",
              "0.5\n",
              "\n",
              "0.6\n",
              "0.7\n",
              "0.8\n",
              "0.9\n",
              "estimated probability of succes\n",
              "\n",
              "1.0\n",
              "\n",
              "0.5\n",
              "\n",
              "(c) Binned distribution for R0 = 1.8.\n",
              "\n",
              "0.6\n",
              "0.7\n",
              "0.8\n",
              "0.9\n",
              "estimated probability of succes\n",
              "\n",
              "1.0\n",
              "\n",
              "(d) Binned distribution for R0 = 2.0.\n",
              "\n",
              "1.0\n",
              "\n",
              "1.0\n",
              "\n",
              "0.9\n",
              "\n",
              "0.9\n",
              "\n",
              "empirical success rate\n",
              "\n",
              "empirical success rate\n",
              "\n",
              "0.8\n",
              "\n",
              "0.8\n",
              "0.7\n",
              "0.6\n",
              "0.5\n",
              "0.4\n",
              "\n",
              "0.8\n",
              "0.7\n",
              "0.6\n",
              "0.5\n",
              "\n",
              "0.5\n",
              "\n",
              "0.6\n",
              "0.7\n",
              "0.8\n",
              "0.9\n",
              "estimated probability of succes\n",
              "\n",
              "1.0\n",
              "\n",
              "0.5\n",
              "\n",
              "(e) Binned distribution for R0 = 2.2.\n",
              "\n",
              "0.6\n",
              "0.7\n",
              "0.8\n",
              "0.9\n",
              "estimated probability of succes\n",
              "\n",
              "1.0\n",
              "\n",
              "(f) Binned distribution for R0 = 2.4.\n",
              "\n",
              "8\n",
              "\n",
              "\fS7\n",
              "\n",
              "Computational resources\n",
              "\n",
              "The simulations were run on a high performance cluster (HPC). On this HPC,\n",
              "we used “Ivy Bridge” nodes, more specifically nodes with two 10-core ”Ivy\n",
              "Bridge” Xeon E5-2680v2 CPUs (2.8 GHz, 25 MB level 3 cache) and 64 GB of\n",
              "RAM. This infrastructure allowed us to run 20 FluTE simulations per node.\n",
              "\n",
              "9\n",
              "\n",
              "\fReferences\n",
              "[1] Abul K Abbas, Andrew H H Lichtman, and Shiv Pillai. Cellular and\n",
              "molecular immunology. Elsevier Health Sciences, 2014.\n",
              "[2] Milton Abramowitz and Irene A Stegun. Handbook of mathematical functions: with formulas, graphs, and mathematical tables, volume 55. Courier\n",
              "Corporation, 1964.\n",
              "[3] Shipra Agrawal and Navin Goyal. Analysis of Thompson sampling for the\n",
              "multi-armed bandit problem. In Conference on Learning Theory, pages\n",
              "31–39, 2012.\n",
              "[4] Maira Aguiar and Nico Stollenwerk. Dengvaxia efficacy dependency on\n",
              "serostatus: a closer look at more recent data. Clinical Infectious Diseases,\n",
              "2017.\n",
              "[5] Sigrún Andradóttir, Wenchi Chiu, David Goldsman, Mi Lim Lee, KwokLeung Tsui, Beate Sander, David N Fisman, and Azhar Nizam. Reactive\n",
              "strategies for containing developing outbreaks of pandemic influenza. BMC\n",
              "public health, 11(1):S1, 2011.\n",
              "[6] Jean-Yves Audibert and Sébastien Bubeck. Best arm identification in\n",
              "multi-armed bandits. In COLT-23th Conference on Learning Theory, 2010.\n",
              "[7] Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of\n",
              "the multiarmed bandit problem. Machine Learning, 47(2-3):235–256, 2002.\n",
              "ISSN 08856125.\n",
              "[8] Nicole E Basta, Dennis L Chao, M Elizabeth Halloran, Laura Matrajt, and\n",
              "Ira M Longini. Strategies for pandemic and seasonal influenza vaccination\n",
              "of schoolchildren in the United States. American journal of epidemiology,\n",
              "170(6):679–686, 2009.\n",
              "[9] Tom Britton. Stochastic epidemic models: a survey. Mathematical biosciences, 225(1):24–35, 2010.\n",
              "[10] Sébastien Bubeck, Rémi Munos, and Gilles Stoltz. Pure exploration in\n",
              "multi-armed bandits problems. In International conference on Algorithmic\n",
              "learning theory, pages 23–37. Springer, 2009.\n",
              "[11] Sébastien Bubeck, Rémi Munos, and Gilles Stoltz. Pure exploration in\n",
              "finitely-armed and continuous-armed bandits. Theoretical Computer Science, 412(19):1832–1852, 2011.\n",
              "[12] Dennis L Chao, M Elizabeth Halloran, Valerie J Obenchain, and Ira M\n",
              "Longini Jr. FluTE, a publicly available stochastic influenza epidemic simulation model. PLoS Computational Biology, 6(1):e1000656, 2010.\n",
              "\n",
              "10\n",
              "\n",
              "\f[13] Dennis L. Chao, Scott B. Halstead, M. Elizabeth Halloran, and Ira M.\n",
              "Longini. Controlling Dengue with Vaccines in Thailand. PLoS Neglected\n",
              "Tropical Diseases, 6(10), 2012. ISSN 19352727.\n",
              "[14] Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. In Advances in neural information processing systems, pages 2249–\n",
              "2257, 2011.\n",
              "[15] Charles J Clopper and Egon S Pearson. The use of confidence or fiducial\n",
              "limits illustrated in the case of the binomial. Biometrika, pages 404–413,\n",
              "1934.\n",
              "[16] Ilaria Dorigatti, Simon Cauchemez, Andrea Pugliese, and Neil Morris Ferguson. A new approach to characterising infectious disease transmission\n",
              "dynamics from sentinel surveillance: application to the Italian 2009/2010\n",
              "A/H1N1 influenza pandemic. Epidemics, 4(1):9–21, 2012.\n",
              "[17] Madalina M. Drugan and Ann Nowe. Designing multi-objective multiarmed bandits algorithms: A study. In Proceedings of the International\n",
              "Joint Conference on Neural Networks, 2013. ISBN 9781467361293.\n",
              "[18] Martin Enserink. Crisis underscores fragility of vaccine production system.\n",
              "Science, 306(5695):385, 2004.\n",
              "[19] S.G. Eubank, V.S.a. Kumar, M.V. Marathe, A. Srinivasan, and N. Wang.\n",
              "Structure of social contact networks and their impact on epidemics. DIMACS Series in Discrete Mathematics and Theoretical Computer Science,\n",
              "70(0208005):181, 2006. ISSN 1052-1798.\n",
              "[20] Stephen G. Eubank, Hasan Guclu, V S Anil Kumar, Madhav V Marathe,\n",
              "Aravind Srinivasan, Zoltan Toroczkai, and Nan Wang. Modelling disease\n",
              "outbreaks in realistic urban social networks. Nature, 429(6988):180–184,\n",
              "2004.\n",
              "[21] Eyal Even-Dar, Shie Mannor, and Yishay Mansour. Action elimination and\n",
              "stopping conditions for the multi-armed bandit and reinforcement learning\n",
              "problems. Journal of machine learning research, 7(Jun):1079–1105, 2006.\n",
              "[22] Neil M Ferguson, Derek A T Cummings, Simon Cauchemez, Christophe\n",
              "Fraser, and Others. Strategies for containing an emerging influenza pandemic in Southeast Asia. Nature, 437(7056):209, 2005.\n",
              "[23] Neil M Ferguson, Isabel Rodrı́guez-Barraquer, Ilaria Dorigatti, Luis Mier-y\n",
              "Teran-Romero, Daniel J Laydon, and Derek A T Cummings. Benefits and\n",
              "risks of the Sanofi-Pasteur dengue vaccine: Modeling optimal deployment.\n",
              "Science, 353(6303):1033–1036, 2016. ISSN 00429686.\n",
              "[24] Christophe Fraser, Derek A T Cummings, Don Klinkenberg, Donald S\n",
              "Burke, and Neil M Ferguson. Influenza transmission in households during\n",
              "the 1918 pandemic. American journal of epidemiology, 174(5):505–514,\n",
              "2011.\n",
              "11\n",
              "\n",
              "\f[25] Laura Fumanelli, Marco Ajelli, Stefano Merler, Neil M. Ferguson, and Simon Cauchemez. Model-Based Comprehensive Analysis of School Closure\n",
              "Policies for Mitigating Influenza Epidemics and Pandemics. PLoS Computational Biology, 12(1), 2016. ISSN 15537358.\n",
              "[26] Aurélien Garivier and Emilie Kaufmann. Optimal best arm identification\n",
              "with fixed confidence. In Conference on Learning Theory, pages 998–1027,\n",
              "2016.\n",
              "[27] T. C. Germann, K. Kadau, I. M. Longini, and C. A. Macken. Mitigation\n",
              "strategies for pandemic influenza in the United States. Proceedings of the\n",
              "National Academy of Sciences, 103(15):5935–5940, 2006. ISSN 0027-8424.\n",
              "[28] Sri Rezeki Hadinegoro, Jose Luis Arredondo-Garcı́a, Maria Rosario Capeding, Carmen Deseda, Tawee Chotpitayasunondh, Reynaldo Dietze,\n",
              "H.I. Hj Muhammad Ismail, Humberto Reynales, Kriengsak Limkittikul,\n",
              "Doris Maribel Rivera-Medina, Huu Ngoc Tran, Alain Bouckenooghe,\n",
              "Danaya Chansinghakul, Margarita Cortés, Karen Fanouillere, Remi Forrat, Carina Frago, Sophia Gailhardou, Nicholas Jackson, Fernando Noriega, Eric Plennevaux, T. Anh Wartel, Betzana Zambrano, and Melanie\n",
              "Saville. Efficacy and Long-Term Safety of a Dengue Vaccine in Regions of\n",
              "Endemic Disease. New England Journal of Medicine, 373(13):1195–1206,\n",
              "2015. ISSN 0028-4793.\n",
              "[29] M Elizabeth Halloran, Ira M Longini, Azhar Nizam, and Yang Yang. Containing bioterrorist smallpox. Science (New York, N.Y.), 298(5597):1428–\n",
              "1432, 2002. ISSN 00368075.\n",
              "[30] M Elizabeth Halloran, Neil M Ferguson, Stephen Eubank, Ira M Longini,\n",
              "Derek A T Cummings, Bryan Lewis, Shufu Xu, Christophe Fraser, Anil\n",
              "Vullikanti, Timothy C Germann, and Others. Modeling targeted layered\n",
              "containment of an influenza pandemic in the United States. Proceedings of\n",
              "the National Academy of Sciences, 105(12):4639–4644, 2008.\n",
              "[31] Matthew Hartfield and Samuel Alizon. Introducing the outbreak threshold\n",
              "in epidemiology. PLoS Pathog, 9(6):e1003277, 2013.\n",
              "[32] Robbins Herbert. Some Aspects of the Sequential Design of Experiments.\n",
              "Bulletin of the American Mathematical Society, 58(5):527–535, 1952. ISSN\n",
              "02730979.\n",
              "[33] Matthew Hoffman, Bobak Shahriari, and Nando Freitas. On correlation and\n",
              "budget constraints in model-based bandit optimization with application to\n",
              "automatic machine learning. In Artificial Intelligence and Statistics, pages\n",
              "365–374, 2014.\n",
              "[34] Junya Honda and Akimichi Takemura. Optimality of Thompson Sampling\n",
              "for Gaussian Bandits Depends on Priors. In AISTATS, pages 375–383,\n",
              "2014.\n",
              "12\n",
              "\n",
              "\f[35] Kevin Jamieson, Matthew Malloy, Robert Nowak, and Sébastien Bubeck.\n",
              "lil’ucb: An optimal exploration algorithm for multi-armed bandits. In Conference on Learning Theory, pages 423–439, 2014.\n",
              "[36] Christopher Jennison, Iain M Johnstone, and Bruce W Turnbull. Asymptotically optimal procedures for sequential adaptive selection of the best of\n",
              "several normal means. Statistical decision theory and related topics III, 2:\n",
              "55–86, 1982.\n",
              "[37] Emilie Kaufmann and Shivaram Kalyanakrishnan. Information complexity\n",
              "in bandit subset selection. In Conference on Learning Theory, pages 228–\n",
              "251, 2013.\n",
              "[38] Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On the complexity of best arm identification in multi-armed bandit models. Journal of\n",
              "Machine Learning Research, 17(1):1–42, 2016.\n",
              "[39] Pieter Libin, Timothy Verstraeten, Kristof Theys, Diederik Roijers, Peter\n",
              "Vrancx, and Ann Nowe. Efficient evaluation of influenza mitigation strategies using preventive bandits. AAMAS 2017 Visionary Papers, Lecture\n",
              "Notes in AI, Volume 10643, page In press, 2017.\n",
              "[40] James O Lloyd-Smith, Sebastian J Schreiber, P Ekkehard Kopp, and\n",
              "Wayne M Getz. Superspreading and the effect of individual variation on\n",
              "disease emergence. Nature, 438(7066):355–359, 2005.\n",
              "[41] Ira M Longini Jr, M Elizabeth Halloran, Azhar Nizam, and Yang Yang.\n",
              "Containing pandemic influenza with antiviral agents. American journal of\n",
              "epidemiology, 159(7):623–633, 2004.\n",
              "[42] Huong Q. McLean, Mark G. Thompson, Maria E. Sundaram, Burney A. Kieke, Manjusha Gaglani, Kempapura Murthy, Pedro A. Piedra,\n",
              "Richard K. Zimmerman, Mary Patricia Nowalk, Jonathan M. Raviotta,\n",
              "Michael L. Jackson, Lisa Jackson, Suzanne E. Ohmit, Joshua G. Petrie,\n",
              "Arnold S. Monto, Jennifer K. Meece, Swathi N. Thaker, Jessie R. Clippard, Sarah M. Spencer, Alicia M. Fry, and Edward A. Belongia. Influenza\n",
              "vaccine effectiveness in the United States during 2012-2013: Variable protection by age and virus type. Journal of Infectious Diseases, 211(10):\n",
              "1529–1540, 2015. ISSN 15376613.\n",
              "[43] Jan Medlock and Alison P Galvani. Optimizing influenza vaccine distribution. Science, 325(5948):1705–1708, 2009. ISSN 0036-8075.\n",
              "[44] Lauren Ancel Meyers, M. E J Newman, Michael Martin, and Stephanie\n",
              "Schrag. Applying network theory to epidemics: Control measures for Mycoplasma pneumoniae outbreaks. Emerging Infectious Diseases, 9(2):204–\n",
              "210, 2003. ISSN 10806040.\n",
              "\n",
              "13\n",
              "\n",
              "\f[45] N. A M Molinari, Ismael R. Ortega-Sanchez, Mark L. Messonnier,\n",
              "William W. Thompson, Pascale M. Wortley, Eric Weintraub, and Carolyn B. Bridges. The annual impact of seasonal influenza in the US: Measuring disease burden and costs. Vaccine, 25(27):5086–5096, 2007. ISSN\n",
              "0264410X.\n",
              "[46] Henry Nicholls. Pandemic influenza: the inside story. PLoS Biol, 4(2):e50,\n",
              "2006. ISSN 1545-7885.\n",
              "[47] K David Patterson and Gerald F Pyle. The geography and mortality of\n",
              "the 1918 influenza pandemic. Bulletin of the History of Medicine, 65(1):4,\n",
              "1991.\n",
              "[48] Catharine Paules and Kanta Subbarao. Influenza. The Lancet, pages –,\n",
              "2017. ISSN 0140-6736.\n",
              "[49] Warren B Powell and Ilya O Ryzhov. Optimal learning, volume 841. John\n",
              "Wiley & Sons, 2012.\n",
              "[50] Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-making. Journal of\n",
              "Artificial Intelligence Research, 48:67–113, 2013. ISSN 10769757.\n",
              "[51] Daniel Russo. Simple bayesian algorithms for best arm identification. In\n",
              "Conference on Learning Theory, pages 1417–1418, 2016.\n",
              "[52] Klaus Stöhr. Influenza: WHO cares. The Lancet infectious diseases, 2(9):\n",
              "517, 2002.\n",
              "[53] R S Sutton and A G Barto. Reinforcement learning: an introduction. 1998.\n",
              "ISBN 0262193981.\n",
              "[54] William R Thompson. On the likelihood that one unknown probability\n",
              "exceeds another in view of the evidence of two samples. Biometrika, 25\n",
              "(3/4):285–294, 1933.\n",
              "[55] Duncan J Watts, Roby Muhamad, Daniel C Medina, and Peter S Dodds.\n",
              "Multiscale, resurgent epidemics in a hierarchical metapopulation model.\n",
              "Proceedings of the National Academy of Sciences of the United States of\n",
              "America, 102(32):11157–11162, 2005.\n",
              "[56] WHO. WHO guidelines on the use of vaccines and antivirals during influenza pandemics. 2004.\n",
              "[57] Lander Willem, Sean Stijven, Ekaterina Vladislavleva, Jan Broeckhove,\n",
              "Philippe Beutels, and Niel Hens. Active Learning to Understand Infectious\n",
              "Disease Models and Improve Policy Making. PLoS Comput Biol, 10(4):\n",
              "e1003563, 2014.\n",
              "\n",
              "14\n",
              "\n",
              "\f[58] Joseph T Wu, Steven Riley, Christophe Fraser, and Gabriel M Leung. Reducing the impact of the next influenza pandemic using household-based\n",
              "public health interventions. PLoS medicine, 3(9):e361, 2006.\n",
              "[59] Wan Yang, Jonathan D Sugimoto, M Elizabeth Halloran, Nicole E Basta,\n",
              "Dennis L Chao, Laura Matrajt, Gail Potter, Eben Kenah, and Ira M\n",
              "Longini. The transmissibility and control of pandemic influenza A (H1N1)\n",
              "virus. Science (New York, N.Y.), 326(2009):729–33, 2009. ISSN 1095-9203.\n",
              "\n",
              "15\n",
              "\n",
              "\f</td>\n",
              "      <td id=\"T_ea679_row0_col1\" class=\"data row0 col1\" >2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n"
            ],
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7ad0bc212110>"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "dataset_train_df_ai.head(1).style.set_properties(**{'text-align': 'left'})"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "nzHiMLVs_THj"
      },
      "source": [
        "Define a list of llama_index documents types:  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "evHKewE_AHrV"
      },
      "outputs": [],
      "source": [
        "articles = [Document(text=content) for content in dataset_train_df_ai[\"text\"]]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QIw3SHLQBcTE",
        "outputId": "3b1947f2-7442-4e50-9e6b-74a4c175f1b0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "'articles' is a list, where each elemnet is of type: <class 'llama_index.core.schema.Document'>\n"
          ]
        }
      ],
      "source": [
        "print(\"'articles' is a list, where each elemnet is of type:\", type(articles[0]))"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "m9c3yTXX_xAm"
      },
      "source": [
        "## LLMs Configurations\n",
        "Configure the LLM via OpenAI's API, and configure the LongLLMLinguaPostprocessor.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 458,
          "referenced_widgets": [
            "a8ddfeecb0b540af813d6a504d2c1b37",
            "7512684f21514ee4882f4645d1736463",
            "5a5ebc9a97b24c04899ba3cd6dd99fd8",
            "ab037e38ee4344f0ba7783609495c8c3",
            "ebee1628d1914ce691b264cb3ab06c6a",
            "65dca0f213fd47188c72b947248df0b0",
            "04c02deeff8446cba37520c2997a3ddb",
            "f194f9efc4604a3ca02894c7b2d5e1ed",
            "60ca044c73f34de5874fdf98387ccb78",
            "dcf4fb0280d34f25b519a261b6f0ea51",
            "4aae1caba1d24aa3a7d51619bd85967f",
            "911bfe955a7743f780296165e98f4c30",
            "4370f8c471d048f09fd0282be4d0adde",
            "187d0787b8e74663b770fbc5b97853f9",
            "c4ada977803b486ab2236c2f0f75938f",
            "e79b2242ab744e9a8246f5b756180baa",
            "07c68d2ac8df4ef7a12604283e375286",
            "0f581af6e8444f3395e8aeb9f3d7be0a",
            "45b7c8c7be874c93860e24286a07dae8",
            "803c057fe1c947c79ec8d53d041284af",
            "2f82da76f0ef4d60b76a2d8300cff688",
            "4f5626ea88bc444189ca799a366f3337",
            "def7accc74be4d96b6de4aa2789e2df1",
            "df4554ba2a91429a80392947af250eef",
            "71daddfe33e14231bad9f3cd5b01f24d",
            "8d7b8e1523374631a01ef57996a75c7e",
            "fd57da350d4245cb8948737386552d89",
            "8b3d76020d6444b49ddc6c97d353b1f9",
            "ef90680d4ed14f2981aa016acc3e259c",
            "32233dee55bc429cbd1d2bce61111f4f",
            "b44b57660416480faa8474ec9bb80eb3",
            "ec58f6571d074322ade8a4c738d1909a",
            "a24d90f5a0804347857799ad38a64b74",
            "50c643e928e34c1391b621e51f4a6358",
            "f8a5e35b66224321a963f00e772811b7",
            "6b8c422fcea247dca531aed9bef156bd",
            "faa5d010a210447fae5bbc0bc5e2d22c",
            "e648d20b5b074b96a36bb39635b387ba",
            "8e918e7d52f8448e80c7b8a53dbf25b6",
            "74a22f4623114c448ef93bf7471ce2b8",
            "87a720e4be6c4439b6ee4a8fcfa7fa1e",
            "efdae790b34e49c395ffb330f6c164c3",
            "5edd8311060c4461afc4fe274c0c02b3",
            "9a6a8de0a1e24f099aef90a1728bcab4",
            "585e97fa62dd4a6996fbb8ef9b7199e1",
            "e1be19d194f14c88ab8233727bac65d9",
            "c61a962f713f42929467982bc9a776ae",
            "bbe9d1c18c2b421abd521f84eea449bc",
            "d37b233523264dcc96128537eadfd733",
            "ef3ec057b92e43348fbcd3d3e550cd03",
            "47be4176b1a84775a1ae92b225e7bc57",
            "04bec435dc1349908cfba1e7d3f3505d",
            "61e7faf6018b47a9a9ab9379fea01203",
            "502796f796ce4f4f9b4edd5249d4659d",
            "3cd67d8eb8264d85a545ca965528c042",
            "6972efd16b6c47da984498c6a7e79c75",
            "c87dc41e732f4e57a00e26924dc37159",
            "f00ad1c12ee241a7948577af64ee07c0",
            "0c96277911dd4928b7ed157d11e4099f",
            "bf11e74f0f50412da3e9ab9b411ca836",
            "917aa0278b3c4223b149cf9ff160a28d",
            "2dac2ac15065467699da7d550fbfdc13",
            "2eebcb977ca14a7ab70527939b40e2bc",
            "6190a6b2ce524554afe5631ff8f535e3",
            "741467e6555c4c0d9ac2ef589b571c7e",
            "92c6791a4d9d430690ab9fe21d34d973",
            "e7061aa0055942f9a0ac1ae190f24dcd",
            "150540c6e0aa4773aa5c9fddb4f891ba",
            "f3bc48dcfae24fd98b13b2772ce4eeb5",
            "281ac210c447468fa9fe86bd3f111b0c",
            "80c5be9af8214b6c80d9d1d92b8a8d94",
            "10f78cc6b74f419c9bb214e278ec560a",
            "f253632485b54ebea17077ff96cf7e99",
            "5a007fc30dbb4faf9514ead9ea0a7ac8",
            "3efb726ffaff43dd917cf5f0b7c38ad3",
            "eb2ad68ee29645f890377070048e2744",
            "28a1ceaa0ae44f54a477b56583142191",
            "87101c3895544aaa86fa0378334b5e6f",
            "83e4434f633b4b0c825ae8c7432bec43",
            "12f96b43866945a2bc470bbc3b58c5f8",
            "576649b4e2ab429a86434d8f09e5bc8b",
            "7f7e3601b5fb4b70ba1d7d20f9970919",
            "19c1a4a246474dde9fe41017eb6d4ad1",
            "04ede2e8e65642d0b2df15c1721ab990",
            "761404f74f034bae8cb60c25f3ee6961",
            "1495569114124cdaace31418533edc71",
            "e53f57c0641144dd86d377de47a3a2f1",
            "36b2443890e34893988753415a584797",
            "f188aa3b023b440c9dec5c618531224d",
            "873833bb256044eca132730227e313c1",
            "02fd3df7568b46b6b95c544a9a831c20",
            "514968c4b88d4e6aa912503a20fa416b",
            "d9f9c9c61a3b4f74a7a2bc774a2429e5",
            "958c9ca1fa3e4fddb7f70b7920800146",
            "4ecaae6ce0f64a20adabe9f21fbf90ae",
            "c1bfec3ebf304e20a29beb15b482fea1",
            "4cad24f7c8ed43e691a8cc88196ef314",
            "0cebbf65e18f4188b8d83e3d3bc4cead",
            "073cd8d70f244a96a7678bbeb02dd133",
            "e1146257fd69407e943620ea4773dd16",
            "ee64c5b33f1640a4ab63c765ff939e65",
            "29902407ba74418d950035dc6f8b0f40",
            "111edf6f7b3345e1a0e09ba81377cee2",
            "d615edfd4e794f12b50d5d76cdde2a61",
            "f3664f6e3b2646688b33e57574cfc17a",
            "8db2bf2131c1432ea5ea7e0df9e25c65",
            "3ca92f216e3f4d199e7682f05da401ea",
            "1fc2f8defe604203a67e4baada02d43d",
            "7f6a00cf2e25424098c0a93add5db934",
            "4906428eb6a84f63aafab0958d889a72",
            "a55a02a2d7da4c3ea4c8f71bfda008d8",
            "3179e189afc3471aa50842ae3617aa3d",
            "236a3785f68c49ebb5f102f37f21d6e5",
            "27ccc1e029184bcfbcb0caaaeea95ae2",
            "5743bf097f8f4826bf3afbc3e16522a1",
            "6f28c1e55db94a9f8ee27ff2479a7082",
            "c914a2c191364003850d97b08fd9d37c",
            "3f97bd7a9e334052b03bcf6ea496903e",
            "78253e26c07b45769c42ad7e3d9d4fd5",
            "5d3dac3b53114117bbb55fb27bc7efba",
            "2b7eedb9cf4841f0b6dd79514deaa460"
          ]
        },
        "id": "lnlerLOEZwvE",
        "outputId": "354681d3-4353-430e-ffed-66f1f5a37db8"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a8ddfeecb0b540af813d6a504d2c1b37",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "config.json:   0%|          | 0.00/583 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "911bfe955a7743f780296165e98f4c30",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "tokenizer_config.json:   0%|          | 0.00/746 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "def7accc74be4d96b6de4aa2789e2df1",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "tokenizer.model:   0%|          | 0.00/500k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "50c643e928e34c1391b621e51f4a6358",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "tokenizer.json:   0%|          | 0.00/1.84M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "585e97fa62dd4a6996fbb8ef9b7199e1",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "special_tokens_map.json:   0%|          | 0.00/435 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6972efd16b6c47da984498c6a7e79c75",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "model.safetensors.index.json:   0%|          | 0.00/26.8k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "e7061aa0055942f9a0ac1ae190f24dcd",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading shards:   0%|          | 0/2 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "87101c3895544aaa86fa0378334b5e6f",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "model-00001-of-00002.safetensors:   0%|          | 0.00/9.98G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "f188aa3b023b440c9dec5c618531224d",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "model-00002-of-00002.safetensors:   0%|          | 0.00/3.50G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "e1146257fd69407e943620ea4773dd16",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a55a02a2d7da4c3ea4c8f71bfda008d8",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "generation_config.json:   0%|          | 0.00/179 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/generation/configuration_utils.py:392: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/transformers/generation/configuration_utils.py:397: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.\n",
            "  warnings.warn(\n"
          ]
        }
      ],
      "source": [
        "llm = OpenAI(model=gpt_type)\n",
        "query = \"Does this publication involve Reinforcement Learning? Answer in a single word, either Yes or No\"\n",
        "\n",
        "postprocessor = LongLLMLinguaPostprocessor(\n",
        "    model_name = \"NousResearch/Llama-2-7b-hf\",\n",
        "    target_token=target_token,\n",
        "    rank_method=\"longllmlingua\",\n",
        "    additional_compress_kwargs={\n",
        "        \"condition_compare\": True,\n",
        "        \"condition_in_question\": \"after\",\n",
        "        \"context_budget\": \"+100\",\n",
        "        \"reorder_context\": \"sort\",\n",
        "        \"dynamic_context_compression_ratio\": 0.3,\n",
        "    },\n",
        ")\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "8Ot5MwoG__ez"
      },
      "source": [
        "## The Experiments\n",
        "Each experiment is independent of the other.  \n",
        "In each experiment we simulate a separate RAG task of asking the LLM to classify a single publication as either discussing a Reinforcement Learning topic, or not.  \n",
        "This task is then deployed in two approaches, the common RAG approach, and the LongLLMLingua RAG approach where the retrieved context is being compressed before sent to the LLM as reference.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kYRAcUZBKT8T",
        "outputId": "83b5cc72-27a6-444e-c23f-b617e3c63027"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "                                                 document_index: 1577\n",
            "                                                 Iteration: 1 / 60\n",
            "------------------------------------------------ Non-compressed response: Yes\n",
            "------------------------------------------------ Compressed response: Yes\n",
            "Original Tokens: 5121\n",
            "Compressed Tokens: 359\n",
            "Comressed Ratio: 14.22x\n",
            "\n",
            "\n",
            "                                                 document_index: 1722\n",
            "                                                 Iteration: 2 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5565\n",
            "Compressed Tokens: 431\n",
            "Comressed Ratio: 12.88x\n",
            "\n",
            "\n",
            "                                                 document_index: 165\n",
            "                                                 Iteration: 3 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: Yes\n",
            "Original Tokens: 5595\n",
            "Compressed Tokens: 419\n",
            "Comressed Ratio: 13.32x\n",
            "\n",
            "\n",
            "                                                 document_index: 1060\n",
            "                                                 Iteration: 4 / 60\n",
            "------------------------------------------------ Non-compressed response: Yes\n",
            "------------------------------------------------ Compressed response: Yes\n",
            "Original Tokens: 4916\n",
            "Compressed Tokens: 326\n",
            "Comressed Ratio: 15.03x\n",
            "\n",
            "\n",
            "                                                 document_index: 2094\n",
            "                                                 Iteration: 5 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 6012\n",
            "Compressed Tokens: 405\n",
            "Comressed Ratio: 14.81x\n",
            "\n",
            "\n",
            "                                                 document_index: 1990\n",
            "                                                 Iteration: 6 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5239\n",
            "Compressed Tokens: 456\n",
            "Comressed Ratio: 11.46x\n",
            "\n",
            "\n",
            "                                                 document_index: 1658\n",
            "                                                 Iteration: 7 / 60\n",
            "------------------------------------------------ Non-compressed response: Yes\n",
            "------------------------------------------------ Compressed response: Yes\n",
            "Original Tokens: 5898\n",
            "Compressed Tokens: 521\n",
            "Comressed Ratio: 11.30x\n",
            "\n",
            "\n",
            "                                                 document_index: 1242\n",
            "                                                 Iteration: 8 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5754\n",
            "Compressed Tokens: 326\n",
            "Comressed Ratio: 17.60x\n",
            "\n",
            "\n",
            "                                                 document_index: 1952\n",
            "                                                 Iteration: 9 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 4794\n",
            "Compressed Tokens: 437\n",
            "Comressed Ratio: 10.95x\n",
            "\n",
            "\n",
            "                                                 document_index: 1466\n",
            "                                                 Iteration: 10 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5595\n",
            "Compressed Tokens: 391\n",
            "Comressed Ratio: 14.27x\n",
            "\n",
            "\n",
            "                                                 document_index: 2389\n",
            "                                                 Iteration: 11 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 6220\n",
            "Compressed Tokens: 354\n",
            "Comressed Ratio: 17.52x\n",
            "\n",
            "\n",
            "                                                 document_index: 894\n",
            "                                                 Iteration: 12 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5913\n",
            "Compressed Tokens: 590\n",
            "Comressed Ratio: 10.01x\n",
            "\n",
            "\n",
            "                                                 document_index: 2067\n",
            "                                                 Iteration: 13 / 60\n",
            "------------------------------------------------ Non-compressed response: Yes\n",
            "------------------------------------------------ Compressed response: Yes\n",
            "Original Tokens: 5083\n",
            "Compressed Tokens: 422\n",
            "Comressed Ratio: 12.02x\n",
            "\n",
            "\n",
            "                                                 document_index: 570\n",
            "                                                 Iteration: 14 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5706\n",
            "Compressed Tokens: 595\n",
            "Comressed Ratio: 9.57x\n",
            "\n",
            "\n",
            "                                                 document_index: 1154\n",
            "                                                 Iteration: 15 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5472\n",
            "Compressed Tokens: 305\n",
            "Comressed Ratio: 17.88x\n",
            "\n",
            "\n",
            "                                                 document_index: 572\n",
            "                                                 Iteration: 16 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5965\n",
            "Compressed Tokens: 404\n",
            "Comressed Ratio: 14.73x\n",
            "\n",
            "\n",
            "                                                 document_index: 388\n",
            "                                                 Iteration: 17 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5170\n",
            "Compressed Tokens: 334\n",
            "Comressed Ratio: 15.43x\n",
            "\n",
            "\n",
            "                                                 document_index: 2532\n",
            "                                                 Iteration: 18 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5690\n",
            "Compressed Tokens: 411\n",
            "Comressed Ratio: 13.81x\n",
            "\n",
            "\n",
            "                                                 document_index: 1026\n",
            "                                                 Iteration: 19 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5082\n",
            "Compressed Tokens: 378\n",
            "Comressed Ratio: 13.41x\n",
            "\n",
            "\n",
            "                                                 document_index: 2181\n",
            "                                                 Iteration: 20 / 60\n",
            "------------------------------------------------ Non-compressed response: Yes\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 6231\n",
            "Compressed Tokens: 187\n",
            "Comressed Ratio: 33.14x\n",
            "\n",
            "\n",
            "                                                 document_index: 2465\n",
            "                                                 Iteration: 21 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5549\n",
            "Compressed Tokens: 354\n",
            "Comressed Ratio: 15.63x\n",
            "\n",
            "\n",
            "                                                 document_index: 601\n",
            "                                                 Iteration: 22 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5338\n",
            "Compressed Tokens: 572\n",
            "Comressed Ratio: 9.32x\n",
            "\n",
            "\n",
            "                                                 document_index: 1270\n",
            "                                                 Iteration: 23 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5466\n",
            "Compressed Tokens: 436\n",
            "Comressed Ratio: 12.51x\n",
            "\n",
            "\n",
            "                                                 document_index: 404\n",
            "                                                 Iteration: 24 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 6302\n",
            "Compressed Tokens: 518\n",
            "Comressed Ratio: 12.14x\n",
            "\n",
            "\n",
            "                                                 document_index: 302\n",
            "                                                 Iteration: 25 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5539\n",
            "Compressed Tokens: 525\n",
            "Comressed Ratio: 10.53x\n",
            "\n",
            "\n",
            "                                                 document_index: 1352\n",
            "                                                 Iteration: 26 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5726\n",
            "Compressed Tokens: 559\n",
            "Comressed Ratio: 10.22x\n",
            "\n",
            "\n",
            "                                                 document_index: 1933\n",
            "                                                 Iteration: 27 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5030\n",
            "Compressed Tokens: 494\n",
            "Comressed Ratio: 10.16x\n",
            "\n",
            "\n",
            "                                                 document_index: 2292\n",
            "                                                 Iteration: 28 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5911\n",
            "Compressed Tokens: 497\n",
            "Comressed Ratio: 11.87x\n",
            "\n",
            "\n",
            "                                                 document_index: 412\n",
            "                                                 Iteration: 29 / 60\n",
            "------------------------------------------------ Non-compressed response: Yes\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5256\n",
            "Compressed Tokens: 467\n",
            "Comressed Ratio: 11.23x\n",
            "\n",
            "\n",
            "                                                 document_index: 1449\n",
            "                                                 Iteration: 30 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5256\n",
            "Compressed Tokens: 457\n",
            "Comressed Ratio: 11.48x\n",
            "\n",
            "\n",
            "                                                 document_index: 1778\n",
            "                                                 Iteration: 31 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5431\n",
            "Compressed Tokens: 417\n",
            "Comressed Ratio: 12.99x\n",
            "\n",
            "\n",
            "                                                 document_index: 1295\n",
            "                                                 Iteration: 32 / 60\n",
            "------------------------------------------------ Non-compressed response: Yes\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5605\n",
            "Compressed Tokens: 414\n",
            "Comressed Ratio: 13.51x\n",
            "\n",
            "\n",
            "                                                 document_index: 2502\n",
            "                                                 Iteration: 33 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5810\n",
            "Compressed Tokens: 532\n",
            "Comressed Ratio: 10.90x\n",
            "\n",
            "\n",
            "                                                 document_index: 837\n",
            "                                                 Iteration: 34 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5790\n",
            "Compressed Tokens: 426\n",
            "Comressed Ratio: 13.56x\n",
            "\n",
            "\n",
            "                                                 document_index: 2263\n",
            "                                                 Iteration: 35 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5348\n",
            "Compressed Tokens: 444\n",
            "Comressed Ratio: 12.02x\n",
            "\n",
            "\n",
            "                                                 document_index: 1953\n",
            "                                                 Iteration: 36 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 4658\n",
            "Compressed Tokens: 541\n",
            "Comressed Ratio: 8.59x\n",
            "\n",
            "\n",
            "                                                 document_index: 1813\n",
            "                                                 Iteration: 37 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5732\n",
            "Compressed Tokens: 363\n",
            "Comressed Ratio: 15.75x\n",
            "\n",
            "\n",
            "                                                 document_index: 2135\n",
            "                                                 Iteration: 38 / 60\n",
            "------------------------------------------------ Non-compressed response: Yes\n",
            "------------------------------------------------ Compressed response: Yes\n",
            "Original Tokens: 5510\n",
            "Compressed Tokens: 414\n",
            "Comressed Ratio: 13.28x\n",
            "\n",
            "\n",
            "                                                 document_index: 1066\n",
            "                                                 Iteration: 39 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 4798\n",
            "Compressed Tokens: 406\n",
            "Comressed Ratio: 11.79x\n",
            "\n",
            "\n",
            "                                                 document_index: 255\n",
            "                                                 Iteration: 40 / 60\n",
            "------------------------------------------------ Non-compressed response: Yes\n",
            "------------------------------------------------ Compressed response: Yes\n",
            "Original Tokens: 5615\n",
            "Compressed Tokens: 460\n",
            "Comressed Ratio: 12.18x\n",
            "\n",
            "\n",
            "                                                 document_index: 2247\n",
            "                                                 Iteration: 41 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5377\n",
            "Compressed Tokens: 485\n",
            "Comressed Ratio: 11.06x\n",
            "\n",
            "\n",
            "                                                 document_index: 57\n",
            "                                                 Iteration: 42 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5829\n",
            "Compressed Tokens: 409\n",
            "Comressed Ratio: 14.22x\n",
            "\n",
            "\n",
            "                                                 document_index: 382\n",
            "                                                 Iteration: 43 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5453\n",
            "Compressed Tokens: 390\n",
            "Comressed Ratio: 13.95x\n",
            "\n",
            "\n",
            "                                                 document_index: 1633\n",
            "                                                 Iteration: 44 / 60\n",
            "------------------------------------------------ Non-compressed response: Yes\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5382\n",
            "Compressed Tokens: 446\n",
            "Comressed Ratio: 12.04x\n",
            "\n",
            "\n",
            "                                                 document_index: 2561\n",
            "                                                 Iteration: 45 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5175\n",
            "Compressed Tokens: 466\n",
            "Comressed Ratio: 11.08x\n",
            "\n",
            "\n",
            "                                                 document_index: 4\n",
            "                                                 Iteration: 46 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 4990\n",
            "Compressed Tokens: 426\n",
            "Comressed Ratio: 11.69x\n",
            "\n",
            "\n",
            "                                                 document_index: 2506\n",
            "                                                 Iteration: 47 / 60\n",
            "------------------------------------------------ Non-compressed response: Yes\n",
            "------------------------------------------------ Compressed response: Yes\n",
            "Original Tokens: 5743\n",
            "Compressed Tokens: 468\n",
            "Comressed Ratio: 12.25x\n",
            "\n",
            "\n",
            "                                                 document_index: 2021\n",
            "                                                 Iteration: 48 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5417\n",
            "Compressed Tokens: 434\n",
            "Comressed Ratio: 12.45x\n",
            "\n",
            "\n",
            "                                                 document_index: 1364\n",
            "                                                 Iteration: 49 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 3382\n",
            "Compressed Tokens: 533\n",
            "Comressed Ratio: 6.33x\n",
            "\n",
            "\n",
            "                                                 document_index: 999\n",
            "                                                 Iteration: 50 / 60\n",
            "------------------------------------------------ Non-compressed response: Yes\n",
            "------------------------------------------------ Compressed response: Yes\n",
            "Original Tokens: 5865\n",
            "Compressed Tokens: 396\n",
            "Comressed Ratio: 14.77x\n",
            "\n",
            "\n",
            "                                                 document_index: 1332\n",
            "                                                 Iteration: 51 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5244\n",
            "Compressed Tokens: 535\n",
            "Comressed Ratio: 9.78x\n",
            "\n",
            "\n",
            "                                                 document_index: 257\n",
            "                                                 Iteration: 52 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5748\n",
            "Compressed Tokens: 396\n",
            "Comressed Ratio: 14.48x\n",
            "\n",
            "\n",
            "                                                 document_index: 782\n",
            "                                                 Iteration: 53 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5721\n",
            "Compressed Tokens: 487\n",
            "Comressed Ratio: 11.72x\n",
            "\n",
            "\n",
            "                                                 document_index: 2324\n",
            "                                                 Iteration: 54 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5080\n",
            "Compressed Tokens: 588\n",
            "Comressed Ratio: 8.62x\n",
            "\n",
            "\n",
            "                                                 document_index: 908\n",
            "                                                 Iteration: 55 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 4895\n",
            "Compressed Tokens: 403\n",
            "Comressed Ratio: 12.12x\n",
            "\n",
            "\n",
            "                                                 document_index: 977\n",
            "                                                 Iteration: 56 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5539\n",
            "Compressed Tokens: 366\n",
            "Comressed Ratio: 15.09x\n",
            "\n",
            "\n",
            "                                                 document_index: 583\n",
            "                                                 Iteration: 57 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5285\n",
            "Compressed Tokens: 340\n",
            "Comressed Ratio: 15.50x\n",
            "\n",
            "\n",
            "                                                 document_index: 2224\n",
            "                                                 Iteration: 58 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5354\n",
            "Compressed Tokens: 476\n",
            "Comressed Ratio: 11.22x\n",
            "\n",
            "\n",
            "                                                 document_index: 1834\n",
            "                                                 Iteration: 59 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5620\n",
            "Compressed Tokens: 507\n",
            "Comressed Ratio: 11.06x\n",
            "\n",
            "\n",
            "                                                 document_index: 373\n",
            "                                                 Iteration: 60 / 60\n",
            "------------------------------------------------ Non-compressed response: No\n",
            "------------------------------------------------ Compressed response: No\n",
            "Original Tokens: 5864\n",
            "Compressed Tokens: 575\n",
            "Comressed Ratio: 10.18x\n"
          ]
        }
      ],
      "source": [
        "# Set a dict to record the outcomes:\n",
        "# The agreements between the two classification RAGs, and the metrics that are being tracked:\n",
        "record = {\"Yes,Yes\": 0,\n",
        "          \"No,No\": 0,\n",
        "          \"Yes,No\": 0,\n",
        "          \"No,Yes\": 0,\n",
        "          \"original_tokens\": [],\n",
        "          \"compressed_tokens\": [],\n",
        "          \"ratios\": [],\n",
        "          \"time_nc\": [],\n",
        "          \"time_c\": [],\n",
        "          }\n",
        "\n",
        "# This is good practice so to clear unnecessary cache space from the GPU:\n",
        "gc.collect()\n",
        "torch.cuda.empty_cache()\n",
        "\n",
        "# Let's choose the publications randomly, since we won't be iterating over all of them:\n",
        "random.seed(set_seed)\n",
        "random_iterations = random.sample(range(len(articles)), num_of_iterations)\n",
        "iteration_counter = 1\n",
        "\n",
        "# Now we iterate over the publication, each iteration is an independent experiment:\n",
        "for document_index in random_iterations:\n",
        "  # Refer only to the relevant document for retrieval, and process it into the vector database:\n",
        "  index = VectorStoreIndex.from_documents(articles[document_index:(document_index + 1)])\n",
        "  print(\"\\n\\n                                                 document_index:\", document_index)\n",
        "  print(\"                                                 Iteration:\", iteration_counter, \"/\", num_of_iterations)\n",
        "  iteration_counter += 1\n",
        "\n",
        "  # RAG retrieval: This part is common to both downstream methods, the non-compressed and the compressed:\n",
        "  retrieval_start_time = time.time()\n",
        "  # Retrieve the top_k chunks from the vector DB:\n",
        "  retriever = index.as_retriever(similarity_top_k=similarity_top_k)\n",
        "  retrieved_context = retriever.retrieve(query)\n",
        "  context_list = [chunk.get_content() for chunk in retrieved_context]\n",
        "  retrieval_time = time.time() - retrieval_start_time\n",
        "\n",
        "\n",
        "  # Vanilla non-compressed RAG approach:\n",
        "  # Prompting the LLM with no compression, the query is supplemented by the retrieved context chunks, i.e. vanilla RAG approach:\n",
        "  start_time_nc = time.time()\n",
        "  prompt = \"\\n\\n\".join(context_list + [query])\n",
        "  response_nc = llm.complete(prompt)\n",
        "  time_nc = time.time() - start_time_nc\n",
        "  print(\"------------------------------------------------ Non-compressed response: \" + str(response_nc))\n",
        "\n",
        "  original_contexts = \"\\n\\n\".join(context_list)\n",
        "  original_tokens = postprocessor._llm_lingua.get_token_length(original_contexts)\n",
        "\n",
        "  # It is necessary to clear GPU memory here:\n",
        "  del prompt\n",
        "  del context_list\n",
        "  del original_contexts\n",
        "  gc.collect()\n",
        "  torch.cuda.empty_cache()\n",
        "\n",
        "  # Compressed context RAG approach:\n",
        "  # Prompting the LLM with the compressed context:\n",
        "  start_time_c = time.time()\n",
        "  new_retrieved_context = postprocessor.postprocess_nodes(retrieved_context,\n",
        "                                                          query_bundle=QueryBundle(query_str=query))\n",
        "\n",
        "  response_c = CompactAndRefine().synthesize(query, new_retrieved_context)\n",
        "\n",
        "\n",
        "  time_c = time.time() - start_time_c\n",
        "\n",
        "\n",
        "  print(\"------------------------------------------------ Compressed response: \" + str(response_c))\n",
        "\n",
        "  compressed_contexts = \"\\n\\n\".join([chunk.get_content() for chunk in new_retrieved_context])\n",
        "\n",
        "  compressed_tokens = postprocessor._llm_lingua.get_token_length(compressed_contexts)\n",
        "  ratio = original_tokens/(compressed_tokens + 1)\n",
        "\n",
        "  # print(compressed_contexts)\n",
        "  # print()\n",
        "  print(\"Original Tokens:\", original_tokens)\n",
        "  print(\"Compressed Tokens:\", compressed_tokens)\n",
        "  print(\"Comressed Ratio:\", f\"{ratio:.2f}x\")\n",
        "\n",
        "  record[str(response_nc).replace(\".\", \"\") + \",\" + str(response_c).replace(\".\", \"\")] += 1\n",
        "  record[\"original_tokens\"].append(original_tokens)\n",
        "  record[\"compressed_tokens\"].append(compressed_tokens)\n",
        "  record[\"ratios\"].append(ratio)\n",
        "  record[\"time_nc\"].append(time_nc + retrieval_time)\n",
        "  record[\"time_c\"].append(time_c + retrieval_time)\n",
        "\n",
        "  # This is good practice so to clear unnecessary cache space from the GPU:\n",
        "  del compressed_contexts\n",
        "  del new_retrieved_context\n",
        "  del retrieved_context\n",
        "  gc.collect()\n",
        "  torch.cuda.empty_cache()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "_QXLx719qtIH"
      },
      "source": [
        "# Analyze Impact of Context Compression: Reduction of Classification Performance vs. Increase of Resources Efficiency"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "9VhFWieArFc8"
      },
      "source": [
        "### Classification Performance\n",
        "Here we measure the impact of the compression of the retrieved context.  \n",
        "We hold everything else constant, meaning, for the same prompt and the same choice of LLM, we check for rate of aggreement between the case of utilizing the context in its original form, vs. compressing it.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J1l44lwIgP36"
      },
      "outputs": [],
      "source": [
        "num_of_iterations_in_practice = len(record[\"original_tokens\"])\n",
        "counts_of_agreements = record[\"Yes,Yes\"] + record[\"No,No\"]\n",
        "counts_of_disagreements = record[\"Yes,No\"] + record[\"No,Yes\"]\n",
        "reduction_in_tokens = round(100*(1 - sum(record[\"compressed_tokens\"]) / sum(record[\"original_tokens\"])))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8Fk6JXc3fVaH",
        "outputId": "92bfea66-445e-493c-9f0f-7432c2232000"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Agreements: 55 out of 60 total cases\n",
            "Disagreements: 5 out of 60 total cases\n",
            "Agreement rate of 92%\n"
          ]
        }
      ],
      "source": [
        "print(\"Agreements:\", str(counts_of_agreements), \"out of\", str(num_of_iterations_in_practice), \"total cases\")\n",
        "print(\"Disagreements:\", str(counts_of_disagreements), \"out of\", str(num_of_iterations_in_practice), \"total cases\")\n",
        "print(f\"Agreement rate of {round(100*counts_of_agreements / num_of_iterations_in_practice)}%\")\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "ZbO4PzAQswvG"
      },
      "source": [
        "### Reduction of Resources: Reduction of sent token translates directly to reduction of $ expenses\n",
        "Note that in our use-case the returned response is a single word, i.e. a single token, thus we don't need to evaluate the reduction of returned tokens, as they remain the same for both RAG cases.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "C9TzcmnKslVd",
        "outputId": "c890efa6-1216-4365-f663-85a5464dbbd0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Non-compressed: Total tokens sent in 60 calls: 327654\n",
            "Compressed:     Total tokens sent in 60 calls: 26473\n",
            "Reduction in tokens: 92%\n",
            "Comressed Ratio: 12.50x\n"
          ]
        }
      ],
      "source": [
        "print(f\"Non-compressed: Total tokens sent in {num_of_iterations_in_practice} calls:\", str(sum(record[\"original_tokens\"])))\n",
        "print(f\"Compressed:     Total tokens sent in {num_of_iterations_in_practice} calls:\", str(sum(record[\"compressed_tokens\"])))\n",
        "print(f\"Reduction in tokens: {reduction_in_tokens}%\")\n",
        "print(\"Comressed Ratio:\", f\"{1 / (1 - reduction_in_tokens/100):.2f}x\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "EXr8cA1byhUf"
      },
      "source": [
        "## Processing Times"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4qKIzdXQyhhC",
        "outputId": "a947bd86-d065-41a2-f1f6-1126e31b3117"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Non-compressed: Total iteration time over 60 calls: 76\n",
            "Compressed:     Total iteration time over 60 calls: 839\n"
          ]
        }
      ],
      "source": [
        "print(f\"Non-compressed: Total iteration time over {num_of_iterations_in_practice} calls:\", str(round(sum(record[\"time_nc\"]))))\n",
        "print(f\"Compressed:     Total iteration time over {num_of_iterations_in_practice} calls:\", str(round(sum(record[\"time_c\"]))))"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "HXTuD13vt3oy"
      },
      "source": [
        "## **Final Conclusion**\n",
        "Under the current choice of setting, hyperparametes, embeddings, and LLM,  \n",
        "The conclusion is:  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fYcudLwWuMCk",
        "outputId": "a1556855-0f92-4e02-8140-31e5079610d3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "While experiencing a drop of 8% in classification agreement, the reduction in expenses due to reduction in takens is 92%.\n",
            "\n",
            "Meaning, for every $100 spent before compression, the new cost will be 8$.\n",
            "*Remember that a disagreement between the two approaches may mean that either one is correct, and it should not be assumed that the non-compressed approach is more reliable, as it is also more sparse.\n",
            "**Remember that the use of the compressor comes with other trade-offs such as employing a local LLM for compression, which requires GPUs.\n"
          ]
        }
      ],
      "source": [
        "reduction_in_agreement = round(100*(record[\"Yes,No\"] + record[\"No,Yes\"]) / num_of_iterations_in_practice)\n",
        "print(f\"While experiencing a drop of {reduction_in_agreement}% in classification agreement, the reduction in expenses due to reduction in takens is {reduction_in_tokens}%.\\n\")\n",
        "print(f\"Meaning, for every $100 spent before compression, the new cost will be {100 - reduction_in_tokens}$.\")\n",
        "print(\"*Remember that a disagreement between the two approaches may mean that either one is correct, and it should not be assumed that the non-compressed approach is more reliable, as it is also more sparse.\")\n",
        "print(\"**Remember that the use of the compressor comes with other trade-offs such as employing a local LLM for compression, which requires GPUs.\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "vY0Y7mAgnOit",
        "outputId": "6c1612d8-15c7-407a-9456-222ffafed72f"
      },
      "outputs": [
        {
          "data": {
            "application/javascript": "\n    async function download(id, filename, size) {\n      if (!google.colab.kernel.accessAllowed) {\n        return;\n      }\n      const div = document.createElement('div');\n      const label = document.createElement('label');\n      label.textContent = `Downloading \"${filename}\": `;\n      div.appendChild(label);\n      const progress = document.createElement('progress');\n      progress.max = size;\n      div.appendChild(progress);\n      document.body.appendChild(div);\n\n      const buffers = [];\n      let downloaded = 0;\n\n      const channel = await google.colab.kernel.comms.open(id);\n      // Send a message to notify the kernel that we're ready.\n      channel.send({})\n\n      for await (const message of channel.messages) {\n        // Send a message to notify the kernel that we're ready.\n        channel.send({})\n        if (message.buffers) {\n          for (const buffer of message.buffers) {\n            buffers.push(buffer);\n            downloaded += buffer.byteLength;\n            progress.value = downloaded;\n          }\n        }\n      }\n      const blob = new Blob(buffers, {type: 'application/binary'});\n      const a = document.createElement('a');\n      a.href = window.URL.createObjectURL(blob);\n      a.download = filename;\n      div.appendChild(a);\n      a.click();\n      div.remove();\n    }\n  ",
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/javascript": "download(\"download_559ce087-ebc3-44cc-9bfc-b0654924f4c9\", \"record.pickle\", 2125)",
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "out = open(\"record.pickle\", \"wb\")\n",
        "pickle.dump(record, out)\n",
        "out.close()\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "-T5HpIgAHMcJ"
      },
      "source": [
        "If this notebook is run on Google Colab, this command will download the `record` dict to your downloads folder.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TbyXM8fi2hhw"
      },
      "outputs": [],
      "source": [
        "if os.getenv(\"COLAB_RELEASE_TAG\"):\n",
        "  from google.colab import files\n",
        "  files.download(\"record.pickle\")"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "00f247c633f244bf9c692ca8262920c8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_380bfec326bf4cf6ab8c48830b7e9b1c",
            "max": 149548650,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0dbfeb79356d466c934748c754d59400",
            "value": 149548650
          }
        },
        "02fd3df7568b46b6b95c544a9a831c20": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c1bfec3ebf304e20a29beb15b482fea1",
            "max": 3500297344,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_4cad24f7c8ed43e691a8cc88196ef314",
            "value": 3500297344
          }
        },
        "04bec435dc1349908cfba1e7d3f3505d": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "04c02deeff8446cba37520c2997a3ddb": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "04ede2e8e65642d0b2df15c1721ab990": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "073cd8d70f244a96a7678bbeb02dd133": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "07c68d2ac8df4ef7a12604283e375286": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0bf9e2088bb14aeba3bc6e2c3441a768": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0c96277911dd4928b7ed157d11e4099f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_741467e6555c4c0d9ac2ef589b571c7e",
            "placeholder": "​",
            "style": "IPY_MODEL_92c6791a4d9d430690ab9fe21d34d973",
            "value": " 26.8k/26.8k [00:00&lt;00:00, 1.98MB/s]"
          }
        },
        "0cebbf65e18f4188b8d83e3d3bc4cead": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0dbfeb79356d466c934748c754d59400": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "0f581af6e8444f3395e8aeb9f3d7be0a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "10f78cc6b74f419c9bb214e278ec560a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "111edf6f7b3345e1a0e09ba81377cee2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7f6a00cf2e25424098c0a93add5db934",
            "placeholder": "​",
            "style": "IPY_MODEL_4906428eb6a84f63aafab0958d889a72",
            "value": " 2/2 [00:56&lt;00:00, 25.99s/it]"
          }
        },
        "12f96b43866945a2bc470bbc3b58c5f8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_761404f74f034bae8cb60c25f3ee6961",
            "max": 9976578928,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_1495569114124cdaace31418533edc71",
            "value": 9976578928
          }
        },
        "1491d6d8b5154e95812c5c971c3b4eee": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "1495569114124cdaace31418533edc71": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "150540c6e0aa4773aa5c9fddb4f891ba": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_10f78cc6b74f419c9bb214e278ec560a",
            "placeholder": "​",
            "style": "IPY_MODEL_f253632485b54ebea17077ff96cf7e99",
            "value": "Downloading shards: 100%"
          }
        },
        "187d0787b8e74663b770fbc5b97853f9": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_45b7c8c7be874c93860e24286a07dae8",
            "max": 746,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_803c057fe1c947c79ec8d53d041284af",
            "value": 746
          }
        },
        "19c1a4a246474dde9fe41017eb6d4ad1": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1e260394422842bfb8abdcc228e52950": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_9e4b2a702825461a9cd1c2efc3028764",
              "IPY_MODEL_5e6ac0a353514669825fe946c20036fc",
              "IPY_MODEL_c8d2af6aa9ca43628b344f6bd488878d"
            ],
            "layout": "IPY_MODEL_85bc1552ef7d4108959ae8f5a638993c"
          }
        },
        "1fc2f8defe604203a67e4baada02d43d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "221abbda138145cd8e46e764dc9b3860": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "236a3785f68c49ebb5f102f37f21d6e5": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3f97bd7a9e334052b03bcf6ea496903e",
            "max": 179,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_78253e26c07b45769c42ad7e3d9d4fd5",
            "value": 179
          }
        },
        "249359bdbfc142d5b722053b820fef52": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "27ccc1e029184bcfbcb0caaaeea95ae2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5d3dac3b53114117bbb55fb27bc7efba",
            "placeholder": "​",
            "style": "IPY_MODEL_2b7eedb9cf4841f0b6dd79514deaa460",
            "value": " 179/179 [00:00&lt;00:00, 11.0kB/s]"
          }
        },
        "281ac210c447468fa9fe86bd3f111b0c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_eb2ad68ee29645f890377070048e2744",
            "placeholder": "​",
            "style": "IPY_MODEL_28a1ceaa0ae44f54a477b56583142191",
            "value": " 2/2 [02:00&lt;00:00, 54.48s/it]"
          }
        },
        "28a1ceaa0ae44f54a477b56583142191": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "29902407ba74418d950035dc6f8b0f40": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3ca92f216e3f4d199e7682f05da401ea",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_1fc2f8defe604203a67e4baada02d43d",
            "value": 2
          }
        },
        "2a409b6ba29e4e1aba794f452f814bde": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_d2ea697878524f34bef022b9b632a775",
              "IPY_MODEL_f559913d5ef0411683357a5ca50885b5",
              "IPY_MODEL_4f5b1c35348c464a8697a34a770f5ffb"
            ],
            "layout": "IPY_MODEL_b6b75247cdf44f76ba775dc3238b3f5a"
          }
        },
        "2aa538f3cc2541cb970dd9bd2f9bb523": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2b7eedb9cf4841f0b6dd79514deaa460": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2dac2ac15065467699da7d550fbfdc13": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2dc2c29422c3437b8859be6598bd6adc": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2eebcb977ca14a7ab70527939b40e2bc": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2f82da76f0ef4d60b76a2d8300cff688": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3179e189afc3471aa50842ae3617aa3d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6f28c1e55db94a9f8ee27ff2479a7082",
            "placeholder": "​",
            "style": "IPY_MODEL_c914a2c191364003850d97b08fd9d37c",
            "value": "generation_config.json: 100%"
          }
        },
        "32233dee55bc429cbd1d2bce61111f4f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "32f6a39d5eef4977a260fc42d0131b15": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3470c362d3dc4b6e96287fcf0610b4ef": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "34cd9b30041c471b8948b110ef6450f1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "3516d6064921470abf3bf20f4a217d9e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2aa538f3cc2541cb970dd9bd2f9bb523",
            "placeholder": "​",
            "style": "IPY_MODEL_3470c362d3dc4b6e96287fcf0610b4ef",
            "value": " 1.71G/1.71G [01:47&lt;00:00, 16.4MB/s]"
          }
        },
        "36b2443890e34893988753415a584797": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "37d75872325f4dc18e0704ae09950429": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fe75bbaa772644ca8965c34c59eee8c0",
            "placeholder": "​",
            "style": "IPY_MODEL_221abbda138145cd8e46e764dc9b3860",
            "value": " 150M/150M [00:10&lt;00:00, 16.2MB/s]"
          }
        },
        "380bfec326bf4cf6ab8c48830b7e9b1c": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3ca92f216e3f4d199e7682f05da401ea": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3cd67d8eb8264d85a545ca965528c042": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "3eae91e857b64b299649f877c70d6eb4": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3efb726ffaff43dd917cf5f0b7c38ad3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "3f97bd7a9e334052b03bcf6ea496903e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "41585a2384394425b0c6dc9e694db127": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "42daf8300eab49ec9a8040d3b0d9ebcf": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4370f8c471d048f09fd0282be4d0adde": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_07c68d2ac8df4ef7a12604283e375286",
            "placeholder": "​",
            "style": "IPY_MODEL_0f581af6e8444f3395e8aeb9f3d7be0a",
            "value": "tokenizer_config.json: 100%"
          }
        },
        "45b7c8c7be874c93860e24286a07dae8": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "47be4176b1a84775a1ae92b225e7bc57": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4906428eb6a84f63aafab0958d889a72": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4aae1caba1d24aa3a7d51619bd85967f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4cad24f7c8ed43e691a8cc88196ef314": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "4cb1db48b6f14bfcbd112b4865e4b287": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "4d2d13b784e14adb9359bc6c71cd613f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_641f9c94ab244fa0b3a4153953976336",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_66b62850c8384e12b16005798149dd73",
            "value": 1
          }
        },
        "4d6afa19e5d34fa7b74f1dc9ee525e2a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_81972485b98040389eb1f99309ea731d",
            "placeholder": "​",
            "style": "IPY_MODEL_91f3807f41ea459981b6c099d76204e1",
            "value": " 3.80k/3.80k [00:00&lt;00:00, 150kB/s]"
          }
        },
        "4e4548e7d87845b8a51e4da9c57d1174": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4ecaae6ce0f64a20adabe9f21fbf90ae": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4f5626ea88bc444189ca799a366f3337": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4f5b1c35348c464a8697a34a770f5ffb": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_82a954c42f3a402c826ab7a473a42cff",
            "placeholder": "​",
            "style": "IPY_MODEL_a05e76f8351c42cb9aa9ab72f78f31fa",
            "value": " 28388/0 [00:24&lt;00:00, 1075.76 examples/s]"
          }
        },
        "502796f796ce4f4f9b4edd5249d4659d": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "50c643e928e34c1391b621e51f4a6358": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_f8a5e35b66224321a963f00e772811b7",
              "IPY_MODEL_6b8c422fcea247dca531aed9bef156bd",
              "IPY_MODEL_faa5d010a210447fae5bbc0bc5e2d22c"
            ],
            "layout": "IPY_MODEL_e648d20b5b074b96a36bb39635b387ba"
          }
        },
        "514968c4b88d4e6aa912503a20fa416b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0cebbf65e18f4188b8d83e3d3bc4cead",
            "placeholder": "​",
            "style": "IPY_MODEL_073cd8d70f244a96a7678bbeb02dd133",
            "value": " 3.50G/3.50G [00:27&lt;00:00, 241MB/s]"
          }
        },
        "53207f366c654487ae2298fd98de8958": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5598ea77bad94eecbcaff771dd0ae52f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5743bf097f8f4826bf3afbc3e16522a1": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "576649b4e2ab429a86434d8f09e5bc8b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e53f57c0641144dd86d377de47a3a2f1",
            "placeholder": "​",
            "style": "IPY_MODEL_36b2443890e34893988753415a584797",
            "value": " 9.98G/9.98G [01:32&lt;00:00, 133MB/s]"
          }
        },
        "585e97fa62dd4a6996fbb8ef9b7199e1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_e1be19d194f14c88ab8233727bac65d9",
              "IPY_MODEL_c61a962f713f42929467982bc9a776ae",
              "IPY_MODEL_bbe9d1c18c2b421abd521f84eea449bc"
            ],
            "layout": "IPY_MODEL_d37b233523264dcc96128537eadfd733"
          }
        },
        "59173acf03c646f4ab987a27e6a62127": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d8261f3ed03b4999a006b531e78619fc",
            "placeholder": "​",
            "style": "IPY_MODEL_4e4548e7d87845b8a51e4da9c57d1174",
            "value": "Generating validation split: "
          }
        },
        "5a007fc30dbb4faf9514ead9ea0a7ac8": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5a5ebc9a97b24c04899ba3cd6dd99fd8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f194f9efc4604a3ca02894c7b2d5e1ed",
            "max": 583,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_60ca044c73f34de5874fdf98387ccb78",
            "value": 583
          }
        },
        "5bc8736baf6c4ce988f13e432b4f4be6": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5d3dac3b53114117bbb55fb27bc7efba": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5dff1bfeee5148a38580ad5d676af044": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5e6ac0a353514669825fe946c20036fc": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_775933ae7b6c40528669a34b25f6eb2d",
            "max": 1681,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c6ca3ceaaa194f0c80a6712f3c57bfc8",
            "value": 1681
          }
        },
        "5edd8311060c4461afc4fe274c0c02b3": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "60ca044c73f34de5874fdf98387ccb78": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "6190a6b2ce524554afe5631ff8f535e3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "61e7faf6018b47a9a9ab9379fea01203": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "63f50891984640139a7eda23892b35f1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_ac8bec4924ea491e8c7629fec7e725a3",
              "IPY_MODEL_8e05142f69e347b0ab6ba16d3cc09cf1",
              "IPY_MODEL_3516d6064921470abf3bf20f4a217d9e"
            ],
            "layout": "IPY_MODEL_854fc31fd2734a96bd5476503228ccef"
          }
        },
        "641f9c94ab244fa0b3a4153953976336": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "651e126cd06847599331d530fc907c25": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "65dca0f213fd47188c72b947248df0b0": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "66b62850c8384e12b16005798149dd73": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "690b3df3217d4e8fa703818a1ae6015d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0bf9e2088bb14aeba3bc6e2c3441a768",
            "placeholder": "​",
            "style": "IPY_MODEL_34cd9b30041c471b8948b110ef6450f1",
            "value": "Downloading data: 100%"
          }
        },
        "6972efd16b6c47da984498c6a7e79c75": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_c87dc41e732f4e57a00e26924dc37159",
              "IPY_MODEL_f00ad1c12ee241a7948577af64ee07c0",
              "IPY_MODEL_0c96277911dd4928b7ed157d11e4099f"
            ],
            "layout": "IPY_MODEL_bf11e74f0f50412da3e9ab9b411ca836"
          }
        },
        "6b357bc0c2f6415085c497ff50e703bc": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6b8c422fcea247dca531aed9bef156bd": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_87a720e4be6c4439b6ee4a8fcfa7fa1e",
            "max": 1842764,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_efdae790b34e49c395ffb330f6c164c3",
            "value": 1842764
          }
        },
        "6f28c1e55db94a9f8ee27ff2479a7082": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6fe8ffee3ea44d7b8f7d74669be4a737": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_690b3df3217d4e8fa703818a1ae6015d",
              "IPY_MODEL_00f247c633f244bf9c692ca8262920c8",
              "IPY_MODEL_37d75872325f4dc18e0704ae09950429"
            ],
            "layout": "IPY_MODEL_5bc8736baf6c4ce988f13e432b4f4be6"
          }
        },
        "70cfb580763d498a869cdcf259a1cd96": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_41585a2384394425b0c6dc9e694db127",
            "placeholder": "​",
            "style": "IPY_MODEL_de15f73ec9884a3ca2758b378d40c6f0",
            "value": " 146M/146M [00:10&lt;00:00, 16.6MB/s]"
          }
        },
        "71daddfe33e14231bad9f3cd5b01f24d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_32233dee55bc429cbd1d2bce61111f4f",
            "max": 499723,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_b44b57660416480faa8474ec9bb80eb3",
            "value": 499723
          }
        },
        "741467e6555c4c0d9ac2ef589b571c7e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "74a22f4623114c448ef93bf7471ce2b8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7512684f21514ee4882f4645d1736463": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_65dca0f213fd47188c72b947248df0b0",
            "placeholder": "​",
            "style": "IPY_MODEL_04c02deeff8446cba37520c2997a3ddb",
            "value": "config.json: 100%"
          }
        },
        "761404f74f034bae8cb60c25f3ee6961": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "775933ae7b6c40528669a34b25f6eb2d": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "78253e26c07b45769c42ad7e3d9d4fd5": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "7a297bb903904363ac481154e9c8600c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8c79455638c5477fbe25eb73bc79b886",
            "placeholder": "​",
            "style": "IPY_MODEL_e8fead7dfc0a4a33a8e2c1e679e32be6",
            "value": " 2500/0 [00:02&lt;00:00, 1148.56 examples/s]"
          }
        },
        "7f0579d35ce943288f79711d07ba3129": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5dff1bfeee5148a38580ad5d676af044",
            "placeholder": "​",
            "style": "IPY_MODEL_e2388d0ffd9d44b181227996e40f1ce8",
            "value": "Downloading data: 100%"
          }
        },
        "7f6a00cf2e25424098c0a93add5db934": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7f7e3601b5fb4b70ba1d7d20f9970919": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "803c057fe1c947c79ec8d53d041284af": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "808ad1d7b29949969d3aa4ea6ad33105": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "80c5be9af8214b6c80d9d1d92b8a8d94": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "81972485b98040389eb1f99309ea731d": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "82a954c42f3a402c826ab7a473a42cff": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "83e4434f633b4b0c825ae8c7432bec43": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_19c1a4a246474dde9fe41017eb6d4ad1",
            "placeholder": "​",
            "style": "IPY_MODEL_04ede2e8e65642d0b2df15c1721ab990",
            "value": "model-00001-of-00002.safetensors: 100%"
          }
        },
        "83e553d816334b8b8019e05130c0abed": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_59173acf03c646f4ab987a27e6a62127",
              "IPY_MODEL_9e8f8ae8e9b34e5b95adeded355cdae8",
              "IPY_MODEL_7a297bb903904363ac481154e9c8600c"
            ],
            "layout": "IPY_MODEL_cf595012852e49e2a538d7cc65c769c5"
          }
        },
        "854fc31fd2734a96bd5476503228ccef": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "85bc1552ef7d4108959ae8f5a638993c": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8615b76ba082478ba99e1d4cddf1b0b2": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "87101c3895544aaa86fa0378334b5e6f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_83e4434f633b4b0c825ae8c7432bec43",
              "IPY_MODEL_12f96b43866945a2bc470bbc3b58c5f8",
              "IPY_MODEL_576649b4e2ab429a86434d8f09e5bc8b"
            ],
            "layout": "IPY_MODEL_7f7e3601b5fb4b70ba1d7d20f9970919"
          }
        },
        "873833bb256044eca132730227e313c1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_958c9ca1fa3e4fddb7f70b7920800146",
            "placeholder": "​",
            "style": "IPY_MODEL_4ecaae6ce0f64a20adabe9f21fbf90ae",
            "value": "model-00002-of-00002.safetensors: 100%"
          }
        },
        "87526647f4e04a1d82bc961e72b26d86": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "87a720e4be6c4439b6ee4a8fcfa7fa1e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8998660271044d47b3d04f0b623bd280": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3eae91e857b64b299649f877c70d6eb4",
            "placeholder": "​",
            "style": "IPY_MODEL_808ad1d7b29949969d3aa4ea6ad33105",
            "value": "Downloading builder script: 100%"
          }
        },
        "8a2d538ace784e62896070d0d5ebf354": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_e2677b3b568d46e1aaddf20f4b4bf9ad",
              "IPY_MODEL_4d2d13b784e14adb9359bc6c71cd613f",
              "IPY_MODEL_acd6967ac6e341f9afe670aa91a7a31e"
            ],
            "layout": "IPY_MODEL_dbc39de7bee64a07b04a013b49a484d3"
          }
        },
        "8b3d76020d6444b49ddc6c97d353b1f9": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8c79455638c5477fbe25eb73bc79b886": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8d7b8e1523374631a01ef57996a75c7e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ec58f6571d074322ade8a4c738d1909a",
            "placeholder": "​",
            "style": "IPY_MODEL_a24d90f5a0804347857799ad38a64b74",
            "value": " 500k/500k [00:00&lt;00:00, 21.9MB/s]"
          }
        },
        "8db2bf2131c1432ea5ea7e0df9e25c65": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8e05142f69e347b0ab6ba16d3cc09cf1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bea914d2e8db47c7ba3c245a0b3b5baa",
            "max": 1708905012,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_1491d6d8b5154e95812c5c971c3b4eee",
            "value": 1708905012
          }
        },
        "8e918e7d52f8448e80c7b8a53dbf25b6": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "90aec7e6cf684a06aea6e48653fa0512": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "911bfe955a7743f780296165e98f4c30": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_4370f8c471d048f09fd0282be4d0adde",
              "IPY_MODEL_187d0787b8e74663b770fbc5b97853f9",
              "IPY_MODEL_c4ada977803b486ab2236c2f0f75938f"
            ],
            "layout": "IPY_MODEL_e79b2242ab744e9a8246f5b756180baa"
          }
        },
        "917aa0278b3c4223b149cf9ff160a28d": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "91f3807f41ea459981b6c099d76204e1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "926d325278334e63ae32ecd81c76ed9c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_8998660271044d47b3d04f0b623bd280",
              "IPY_MODEL_9a593469841c4ad798590b092d48184a",
              "IPY_MODEL_4d6afa19e5d34fa7b74f1dc9ee525e2a"
            ],
            "layout": "IPY_MODEL_53207f366c654487ae2298fd98de8958"
          }
        },
        "92c6791a4d9d430690ab9fe21d34d973": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "944fe5ee5d8b4fd69df3bf1900d1a578": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "958c9ca1fa3e4fddb7f70b7920800146": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "95cd0639a0ae4f10b445a2b9d6885d4a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9a593469841c4ad798590b092d48184a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_32f6a39d5eef4977a260fc42d0131b15",
            "max": 3798,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_bcf0b1dc6aa44f0ba60865191490ad6a",
            "value": 3798
          }
        },
        "9a6a8de0a1e24f099aef90a1728bcab4": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9e4b2a702825461a9cd1c2efc3028764": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_90aec7e6cf684a06aea6e48653fa0512",
            "placeholder": "​",
            "style": "IPY_MODEL_6b357bc0c2f6415085c497ff50e703bc",
            "value": "Downloading readme: 100%"
          }
        },
        "9e8f8ae8e9b34e5b95adeded355cdae8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4cb1db48b6f14bfcbd112b4865e4b287",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c78397492dfd4bae8dfcd0f26a7e5594",
            "value": 1
          }
        },
        "a05e76f8351c42cb9aa9ab72f78f31fa": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a24d90f5a0804347857799ad38a64b74": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a55a02a2d7da4c3ea4c8f71bfda008d8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3179e189afc3471aa50842ae3617aa3d",
              "IPY_MODEL_236a3785f68c49ebb5f102f37f21d6e5",
              "IPY_MODEL_27ccc1e029184bcfbcb0caaaeea95ae2"
            ],
            "layout": "IPY_MODEL_5743bf097f8f4826bf3afbc3e16522a1"
          }
        },
        "a8ddfeecb0b540af813d6a504d2c1b37": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_7512684f21514ee4882f4645d1736463",
              "IPY_MODEL_5a5ebc9a97b24c04899ba3cd6dd99fd8",
              "IPY_MODEL_ab037e38ee4344f0ba7783609495c8c3"
            ],
            "layout": "IPY_MODEL_ebee1628d1914ce691b264cb3ab06c6a"
          }
        },
        "ab037e38ee4344f0ba7783609495c8c3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_dcf4fb0280d34f25b519a261b6f0ea51",
            "placeholder": "​",
            "style": "IPY_MODEL_4aae1caba1d24aa3a7d51619bd85967f",
            "value": " 583/583 [00:00&lt;00:00, 43.7kB/s]"
          }
        },
        "ac8bec4924ea491e8c7629fec7e725a3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5598ea77bad94eecbcaff771dd0ae52f",
            "placeholder": "​",
            "style": "IPY_MODEL_e1568699381d410fb5112cfb788ba6a4",
            "value": "Downloading data: 100%"
          }
        },
        "acd6967ac6e341f9afe670aa91a7a31e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d915b903dffd47cda0e9fa95907859da",
            "placeholder": "​",
            "style": "IPY_MODEL_42daf8300eab49ec9a8040d3b0d9ebcf",
            "value": " 2500/0 [00:01&lt;00:00, 1335.37 examples/s]"
          }
        },
        "adeb62cdecb649c0b81ed530f0dc58cb": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b2e6c1a987104f1cbab55a1cf965c369": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "b44b57660416480faa8474ec9bb80eb3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "b6b75247cdf44f76ba775dc3238b3f5a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bbe9d1c18c2b421abd521f84eea449bc": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_502796f796ce4f4f9b4edd5249d4659d",
            "placeholder": "​",
            "style": "IPY_MODEL_3cd67d8eb8264d85a545ca965528c042",
            "value": " 435/435 [00:00&lt;00:00, 32.2kB/s]"
          }
        },
        "bcf0b1dc6aa44f0ba60865191490ad6a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "bea914d2e8db47c7ba3c245a0b3b5baa": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bf11e74f0f50412da3e9ab9b411ca836": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c1bfec3ebf304e20a29beb15b482fea1": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c4ada977803b486ab2236c2f0f75938f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2f82da76f0ef4d60b76a2d8300cff688",
            "placeholder": "​",
            "style": "IPY_MODEL_4f5626ea88bc444189ca799a366f3337",
            "value": " 746/746 [00:00&lt;00:00, 52.1kB/s]"
          }
        },
        "c61a962f713f42929467982bc9a776ae": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_04bec435dc1349908cfba1e7d3f3505d",
            "max": 435,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_61e7faf6018b47a9a9ab9379fea01203",
            "value": 435
          }
        },
        "c6ca3ceaaa194f0c80a6712f3c57bfc8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c78397492dfd4bae8dfcd0f26a7e5594": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c87dc41e732f4e57a00e26924dc37159": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_917aa0278b3c4223b149cf9ff160a28d",
            "placeholder": "​",
            "style": "IPY_MODEL_2dac2ac15065467699da7d550fbfdc13",
            "value": "model.safetensors.index.json: 100%"
          }
        },
        "c8d2af6aa9ca43628b344f6bd488878d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8615b76ba082478ba99e1d4cddf1b0b2",
            "placeholder": "​",
            "style": "IPY_MODEL_e35c9154ed09420988dfeb729f17368c",
            "value": " 1.68k/1.68k [00:00&lt;00:00, 117kB/s]"
          }
        },
        "c914a2c191364003850d97b08fd9d37c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "cc46e93ca07d42bab9ae9c4bb14e706d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_95cd0639a0ae4f10b445a2b9d6885d4a",
            "max": 146035962,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_249359bdbfc142d5b722053b820fef52",
            "value": 146035962
          }
        },
        "cf595012852e49e2a538d7cc65c769c5": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d2ea697878524f34bef022b9b632a775": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_651e126cd06847599331d530fc907c25",
            "placeholder": "​",
            "style": "IPY_MODEL_2dc2c29422c3437b8859be6598bd6adc",
            "value": "Generating train split: "
          }
        },
        "d37b233523264dcc96128537eadfd733": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d615edfd4e794f12b50d5d76cdde2a61": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d8261f3ed03b4999a006b531e78619fc": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d915b903dffd47cda0e9fa95907859da": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d9f9c9c61a3b4f74a7a2bc774a2429e5": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dbc39de7bee64a07b04a013b49a484d3": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dcf4fb0280d34f25b519a261b6f0ea51": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "de15f73ec9884a3ca2758b378d40c6f0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "de97f327e21a42e4b077f393f5195183": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "def7accc74be4d96b6de4aa2789e2df1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_df4554ba2a91429a80392947af250eef",
              "IPY_MODEL_71daddfe33e14231bad9f3cd5b01f24d",
              "IPY_MODEL_8d7b8e1523374631a01ef57996a75c7e"
            ],
            "layout": "IPY_MODEL_fd57da350d4245cb8948737386552d89"
          }
        },
        "df4554ba2a91429a80392947af250eef": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8b3d76020d6444b49ddc6c97d353b1f9",
            "placeholder": "​",
            "style": "IPY_MODEL_ef90680d4ed14f2981aa016acc3e259c",
            "value": "tokenizer.model: 100%"
          }
        },
        "e1146257fd69407e943620ea4773dd16": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_ee64c5b33f1640a4ab63c765ff939e65",
              "IPY_MODEL_29902407ba74418d950035dc6f8b0f40",
              "IPY_MODEL_111edf6f7b3345e1a0e09ba81377cee2"
            ],
            "layout": "IPY_MODEL_d615edfd4e794f12b50d5d76cdde2a61"
          }
        },
        "e1568699381d410fb5112cfb788ba6a4": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e1be19d194f14c88ab8233727bac65d9": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ef3ec057b92e43348fbcd3d3e550cd03",
            "placeholder": "​",
            "style": "IPY_MODEL_47be4176b1a84775a1ae92b225e7bc57",
            "value": "special_tokens_map.json: 100%"
          }
        },
        "e2388d0ffd9d44b181227996e40f1ce8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e2677b3b568d46e1aaddf20f4b4bf9ad": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_adeb62cdecb649c0b81ed530f0dc58cb",
            "placeholder": "​",
            "style": "IPY_MODEL_87526647f4e04a1d82bc961e72b26d86",
            "value": "Generating test split: "
          }
        },
        "e35c9154ed09420988dfeb729f17368c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e53f57c0641144dd86d377de47a3a2f1": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e648d20b5b074b96a36bb39635b387ba": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e7061aa0055942f9a0ac1ae190f24dcd": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_150540c6e0aa4773aa5c9fddb4f891ba",
              "IPY_MODEL_f3bc48dcfae24fd98b13b2772ce4eeb5",
              "IPY_MODEL_281ac210c447468fa9fe86bd3f111b0c"
            ],
            "layout": "IPY_MODEL_80c5be9af8214b6c80d9d1d92b8a8d94"
          }
        },
        "e74052b5d9f54b69838fb4e0de72eccd": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_7f0579d35ce943288f79711d07ba3129",
              "IPY_MODEL_cc46e93ca07d42bab9ae9c4bb14e706d",
              "IPY_MODEL_70cfb580763d498a869cdcf259a1cd96"
            ],
            "layout": "IPY_MODEL_944fe5ee5d8b4fd69df3bf1900d1a578"
          }
        },
        "e79b2242ab744e9a8246f5b756180baa": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e8fead7dfc0a4a33a8e2c1e679e32be6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "eb2ad68ee29645f890377070048e2744": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ebee1628d1914ce691b264cb3ab06c6a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ec58f6571d074322ade8a4c738d1909a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ee64c5b33f1640a4ab63c765ff939e65": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f3664f6e3b2646688b33e57574cfc17a",
            "placeholder": "​",
            "style": "IPY_MODEL_8db2bf2131c1432ea5ea7e0df9e25c65",
            "value": "Loading checkpoint shards: 100%"
          }
        },
        "ef3ec057b92e43348fbcd3d3e550cd03": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ef90680d4ed14f2981aa016acc3e259c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "efdae790b34e49c395ffb330f6c164c3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "f00ad1c12ee241a7948577af64ee07c0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2eebcb977ca14a7ab70527939b40e2bc",
            "max": 26788,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_6190a6b2ce524554afe5631ff8f535e3",
            "value": 26788
          }
        },
        "f188aa3b023b440c9dec5c618531224d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_873833bb256044eca132730227e313c1",
              "IPY_MODEL_02fd3df7568b46b6b95c544a9a831c20",
              "IPY_MODEL_514968c4b88d4e6aa912503a20fa416b"
            ],
            "layout": "IPY_MODEL_d9f9c9c61a3b4f74a7a2bc774a2429e5"
          }
        },
        "f194f9efc4604a3ca02894c7b2d5e1ed": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f253632485b54ebea17077ff96cf7e99": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f3664f6e3b2646688b33e57574cfc17a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f3bc48dcfae24fd98b13b2772ce4eeb5": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5a007fc30dbb4faf9514ead9ea0a7ac8",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3efb726ffaff43dd917cf5f0b7c38ad3",
            "value": 2
          }
        },
        "f559913d5ef0411683357a5ca50885b5": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b2e6c1a987104f1cbab55a1cf965c369",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_de97f327e21a42e4b077f393f5195183",
            "value": 1
          }
        },
        "f8a5e35b66224321a963f00e772811b7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8e918e7d52f8448e80c7b8a53dbf25b6",
            "placeholder": "​",
            "style": "IPY_MODEL_74a22f4623114c448ef93bf7471ce2b8",
            "value": "tokenizer.json: 100%"
          }
        },
        "faa5d010a210447fae5bbc0bc5e2d22c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5edd8311060c4461afc4fe274c0c02b3",
            "placeholder": "​",
            "style": "IPY_MODEL_9a6a8de0a1e24f099aef90a1728bcab4",
            "value": " 1.84M/1.84M [00:00&lt;00:00, 2.54MB/s]"
          }
        },
        "fd57da350d4245cb8948737386552d89": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fe75bbaa772644ca8965c34c59eee8c0": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
